ruby on rails - Strategy for prioritizing data updates and maintenance in a data-heavy app? -

i've got rails app tens of thousands of records poll api updates on. i'm wondering what's best practices in terms of creating system (more involved cron job) tracks how record should updated...some records more important/timely others...so regular maintenance, want more important records polled , updated twice day , compared once day non-important records.

so assuming these records have method returns "importance"...should create type of maintenance_record model belongs_to each record , keeps track of record's staleness (a combination of record's updated_at , importance)? record log whether latest attempt successful or not, , perhaps have foreign key table of log records.

the purpose have maintenance_record quick index , sort cron-like job can scan list jobs do, rather hitting records database (which may contain records lots of blobs, etc. , may grow orders of magnitude). , of course, have more 1 kind of record-like model...so having polymorphic maintenance_record seems sensible.

anyway, idea? seems scenario come wondering if there libraries may deal it, though imagine creating simple 1 isn't hard.

in general, when i've needed periodically update records, add modified_at timestamp field gets updated each time record changes.

then, on cron schedule, through table records beyond time, process , update row , modified_at field.

if there priorities/importance, add field , set level each record. if 1 top , 2 next lower, every cycle process fields priority == 1. every other cycle process records priority <= 2.

how should process both fields left exercise you.