Hi Robert, Thanks a lot for the update and good look with the review. Moritz
On 3 Oct 2019, at 11:45, Robert Kisteleki <robert@ripe.net> wrote:
On 2019-10-03 08:16, Moritz Muller wrote:
Hi,
In our experiment we’re trying to assign certain probes to a ping measurement but until now we always get the error message "NO SUITABLE PROBES”. According to the documentation, this is a sign that a probe might not have enough resources. However, when I check on the measurement a few hours later I do see results, but its state is still "NO SUITABLE PROBES”. See https://atlas.ripe.net/measurements/23016956/#!probes
Is that a common problem when selecting certain probes for measurements?
Moritz
Hello,
Yesterday afternoon we had an operational problem within RIPE Atlas that had consequences visible to users. I strongly suspect the above is a side-effect of this.
Due to a combination of two configuration errors and a spike in requests from users, the core infrastructure received an unreasonably high amount of measurement requests from an internal process related to IPmap. The measurements scheduler and participant management subsystems struggled to keep up with this load and eventually things started piling up.
The issue started at approximately 12 UTC. We identified the root cause about an hour later. Processes started to normalise late in the afternoon, and processing the backlog finished sometime after midnight.
We're working on a post-mortem, and reviewing the code and configuration in order to prevent this error from happening again.
Apologies for the inconvenience, Robert