As a fellow Odo-nut, I feel you. sometimes I think that they live to make you doubt your ability to pick an f-stop. 
I’ll give my two cents as someone actually coming from the opposite direction- first camera (analog and digital) and smartphone later.
To find out if you really need a camera system, I would think about what your smartphone currently “lacks” in terms of photos for more observations that you want to cover with it.
In my experience, those are those items:
- zoom (most of the time you are limited to 5x optical for really good models like Pixel, the digital one is crappy for birds far away)
- macro (depending on your model - some phones have a really good one, some are crappy)
- autofocus (getting some critter close to you sharp can be a real pain if you and the phone have a different aesthetic on what should be the motive of the foto…)
Macro (and autofocus) is something you need for close and small objects (insects, plants, mushrooms). Zoom is necessary if you want to take pictures of something that will fly/run away (animals, in particular birds, insects).
There are options for the smartphone for this- clip on lenses for macro or binoculars for zoom, as mentioned above. I don’t have any experience on that.
For my experience in camera systems: I started out with small analog cameras, changed to digital in the early days, had a bridge camera and am now on my current workhorse since over 15 years: a Nikon DLSR + 18- 200 mm (18-400 mm for safaris) zoom + a Google Pixel Pro since two years. The reason for this choice: it fits nearly situation and is still “light” enough that I can take it on long hikes or airplane travel. The only disadvantage is the really crappy macro distance (half a meter for close ups). I fix this by using my smartphone for macros or I finally bought a macro lens last year.
So coming from this, I would also say that I would never recommend a tele fixed lens or a macro lens for a beginner that has no experience with a camera before. The frustration in finding the motive at distance or getting sharp pictures can be really disheartening when coming from a smartphone “where everything works”. As mentionend, I bought (now two) macro lenses for my system just a few months ago - first a 40 mm and now a 105 mm and my learning curve even with my experience on my system has been really frustrating. So many blurry, unsharp pictures and with the 105 mm, finding an insect or a flower in distance and getting it focussed really need experience and time, time, time. As a beginner, I would have thrown the towel long ago.
So my recommendation would be, if you feel the need for a camera outside your smartphone, would be either a DLSR/Mirrorless camera with a zoom like 18-200 mm or more (it’s called a travel zoom) or a bridge camera. I know many others that are into birding that are really happy with their bridges (they also seem to have good macros), but my personal experience with them is ancient. You mentioned birds and insects, so that might be a good fit.
When buying a new system, those are the points I would think about (the importance depends on what you want to observe and what your habit is on how you are observing)
- budget - used or dying out systems like DLSR can be cheeper, but may not have the last technological advantages, if that is important
- weight (!) - those fixed zoom lenses can weight a ton, perhaps you even need a tripod or other equipment or several lenses. If you observe while hiking, you feel every additional gramm (ask me how I know …)
- autofocus - I love that I can fix my focus on a selected small spot. This helps when my camera and I are fighting on whether that nice background or the ugly brown mushroom in the front is a nice picture. Manual focus is not an option for me since either my flowers are shaking in the wind or the insect/animal just won’t hold still. Image stabilizer is a must have when using a long zoom
- macro distance- travel zooms unfortunately have usually a quite bad one, as mentioned. A bridge might serve better here or a macro lens (or simply your smartphone). I know some are also using a macro clip on lens on their camera/standard lens
- zoom - I wouldn’t recommend a fixed length since it takes really some practice to find e.g a bird in a tree and lens shaking can be an issue (where you might need a tripod). A tele zoom lens or bridge lets you zoom in from a wider frame, which might make it easier to find that bird. If you have enough mega pixels, you can also zoom in digitally for a bit (I use that often). Another advantage of using a smaller chip camera like APC or MFT is that you gain zoom (it’s called a crop factor), meaning you suddenly have 1.5 x more zoom compared to what is written on the lens. Bridge cameras are I think the kings of zoom since they have 60x or more depending on models- hence why they are loved by birders
- what I really love about my camera and would never want to miss is the 360 degree movable display + live view auto focus. It means I never have to kneel or lie on the ground but can take shots up to a mushroom underside from standing position (good for bad knees or lazy people)
Hope that helps!
For innating I would steer away from dlsr and go to a bridge camera. Bridge cameras have extreme zoom for birds while maintaining decent close up performance. If you get a camera and a 300 lens you are going to get trouble trying to take insects or plants photos. I would recomend a nikon coolpix P950. The biggest downside is bad low light performance. Almost all the photos on my profile are from a bridge camera “panasonic fz82” or my phone, if you want to snoop around.
If you want a fancy camera, you’re welcome to get one. However, don’t feel obligated or pressured into doing so. I have an iPhone 16 pro and attachable moment 10x and 75x macro lenses which have been great for photographing tiny bugs and flowers and are probably a lot more budget-friendly.
I’ll second the comment from @jrp01. I have both a ‘bridge’ camera (an old Canon Powershot SX60) and a DSLR camera with a range of lenses. Unless I am doing scenic or other photography, that I might want to print in a very large format, I invariably take the bridge camera into the field.
It is light-weight, I don’t have to carry extra lenses with me (and don’t have worry about having to switch lenses, while meanwhile the animal disappears), less expensive, and the optical zoom range is equivalent to DSLR lenses from 21mm to 1365mm. The zoom range has allowed me to do everything from close-ups of small insects to clearly indentifiable shots of eagles flying over 1/4 mile away. 99%+ of the photos in my iNat account have been taken with this ‘bridge’ camera, if you want to get an idea of what it can do.
I went straight from a Fujifilm X-A5 to a Nikon Z9 so I can’t say much about camera quality, but what I can say is that I think starting off with a 28-400mm lens was great. It takes acceptable telephotos at long range and it produces acceptable quality photos of insects when mothing. It was a great lens for starting out because it’s pretty decent at everything I’d want. I’ve since added a 180-600mm lens and I hope to buy more lenses next year, but I still use the 28-400mm 99% of the time simply because it’s light, portable, and it’s alright at everything.