Bsrgvty

It was installed incorrectly.

It appears what you have in the picture is a knotting anchor. Near the tip of the plastic sleeve is a threaded section. Properly installed, tightening the screw should pull the far end of the sleeve up against the wall, making a thick "knot" that prevents the anchor from pulling through the wall. Since your failed anchor looks smooth and undeformed, this evidently did not happen.

Here's an example from NPA:

You say there are wooden studs where the anchors were installed. Perhaps you drilled a hole large enough for the anchor then pushed it into the hole in the stud. Since the body of the anchor was then constrained by the surrounding wood rather than in the empty space behind the wall it was unable to expand. Possibly you also did not tighten the screws with sufficient force.

When mounting something where there is a wooden stud, the proper procedure is to not use an anchor at all, but just run a wood screw into the stud. The holding power of a wood screw in wood is a lot more than any hollow wall anchor.

TL;DR If you mount the shelf supports directly to studs, do that. Just make sure to use long screws - at least 1" into the studs.

There are two typical ways to attach a shelf (or heavy mirror or whatever) to a wall with screws:

Anchors

Anchors are the "plastic things". They come in various sizes/strengths - 30 lb. rating is typical. The way you normally use them is with a hollow wall. In other words, you don't use them if you are mounting over a stud. You drill a hole - or normally I just poke a hole with a screwdriver if the wall is made of drywall - in the wall slightly smaller than the anchor. Then push the anchor in, narrow end first, so that the wide end is flush with the wall surface. Place the shelf support over the anchor and drive the supplied screw through the support and into the anchor. The anchor will expand slightly, hopefully providing a tight grip on the drywall.

A variant is a toggle bolt, where a part goes through the wall, flips/expands and is pulled to the inside of the wall when you put in the screw from the outside. I prefer toggle bolts as they are a bit stronger than typical anchors.

You actually should not use an anchor (and impossible to use a toggle bolt) if you are mounting right over a stud. If you use a power drill then you could make enough of a hole in the stud for the anchor, but it really won't hold very well and it is not designed to be used that way.

Screws Into Studs

The gold standard is long screws into studs. With wood studs (metal is also possible but a little different), you typically drill a pilot hole through drywall and into the stud. Then hold the shelf support over the hole and drive a long screw through the support into the stud. Typically I would use 2" - 3" screws. You don't want to use screws that are too long in case there are wires running through the middle of the studs, but you do want to get around an inch or so into the stud, plus 1/2" for drywall plus ???? for the shelf support - so 2" is typical.

With screws into 2 studs, a shelf or cabinet can support a LOT of weight. But if you can only get into 1 stud due to factors beyond your control, then you can either:

• Use screws into one stud and use anchors for the other support(s). This will stabilize the shelf quite a bit compared to anchors alone.


• Mount a strip of wood horizontally into 2 studs and then use wood screws to attach the shelf supports to the wood. This will also normally be quite strong.

Manassehkatz's answer provides good information. However looking at your photo, it appears you have some type of plastic expansion anchor of some sort.

Because it's now out of the wall, I expect that you did not screw it down hard enough.
That type of anchor should be tightened down so the anchor components are compressed outwards behind the wallboard and prevent the anchor from being pulled back through the hole. Depending on the type, you can try that outside the wall to get a feel for how it's supposed to work.

Kind of like this: https://www.homedepot.com/p/The-Hillman-Group-3-8-in-Pop-Toggle-with-Screw-6-Pack-376255/202243047

I don't think I saw this point mentioned above -

but if you did in fact drill a large, anchor sized hole into the stud, then there is one more way you could recover the hole and use it with wood screws - back fill it.

Purchase a small wooden dowel of approximately the same size as the hole, use wood glue to securely attach it, and when the glue dries, screw a regular wood screw in.

This is a trick used to repair guitar strap pins when the screw strips out of the guitar body, but for that we use toothpicks and wood glue as the diameter is very small. Toothpicks might work here, too.

Jade Steffen offered one solution (back-filling the stud) to rehanging, but I don't see much here to answer that part of your question.

If there are indeed wooden studs behind the holes, those holes are now far too big for any screw you can use with that shelf hanger. Luckily, E-Z Ancor makes a stud-compatible anchor called the "Stud Solver" that should be large enough to grip the stud firmly and provide you a compatible hole to screw into:

https://www.homedepot.com/p/E-Z-Ancor-Stud-Solver-7-x-1-1-4-in-Alloy-Flat-Head-Self-Drilling-Drywall-Anchors-with-Screws-4-Pack-29503/100234590

If there is just drywall there, you'll want a stronger style of anchor like a toggle. You can get the old school style that are just a toggle anchor on a bolt, but I'm personally a fan of the newer strap toggles. I think they're easier to install, and they're extremely strong as drywall anchors go:

https://www.homedepot.com/p/The-Hillman-Group-1-4-in-x-2-1-2-in-Toggle-Straps-with-Screws-6-Pack-376260/202982713

Of course, the best choice might be to patch the holes and relocate the shelf a bit, hanging with screws directly into the studs or attaching a backing board to the studs and mounting the shelf to that. The details for that depend on your space and the aesthetics you're going for. If this is a rental, you might be best off to just patch it, paint it, and get a free-standing bookshelf instead.