We’ve all been there. You order something you need ASAP — diapers, a gift for a friend, shoes for a date — and the company doesn’t get it to you within the time window they guaranteed. You’re pissed, and may end up buying the item at the store yourself in desperation.

Of course, you can call the company and complain, but that takes time and more patience than many of us are willing to invest. That’s why, when Moneyish got an early look at Paribus’ new automatic delivery monitoring and refund feature, we were intrigued.

The free Paribus site and app — which scans your email to look for receipts, checks for price drops on those items from retailers and then automatically files a claim with the company to get you the price difference — has been around a few years. But on Thursday, Paribus, which is now owned by Capital One, is adding a new feature that “automatically files a claim for users if their online orders from select retailers are delivered later than promised.”

The delivery monitoring service works like this: It checks your email for receipts, looks for guaranteed delivery dates and then tracks the package using UPS or the retailer’s own shipping tracking. If the package is late getting to your door, Paribus automatically files a claim with the retailer to get you compensation.

So what kind of compensation might you be entitled to? Amazon says it will “provide a refund of any shipping fees associated with the order in question. Prime members may also be eligible for an extension of their membership” (typically this is a one-month extension). And many other large retailers will refund all or part of the cost of shipping if a product is late getting to you. Some even offer some amount of store credit as compensation.

Paribus, which currently has 1.5 million users, makes money on its price refund feature (it takes a 25% cut of the refund), but is offering the delivery feature to users at no cost. It has dozens of retail partners including Amazon, Walmart, Best Buy and Staples.

Of course, you can call a retailer yourself for a refund without using Paribus, and the app by no means covers all retailers. Plus, late delivery isn’t as common as you might think: In Paribus’ internal case study, only about 4% of packages were late. And you can’t expect a refund on all packages that take a while – refunds typically only apply to situations in which a retailer guaranteed you delivery within a certain window.