For teams building AI
Human evaluation for your models.
IdealJob takes on LLM evaluation and data validation for teams building AI. People we checked on the task type rate your model's outputs and check your labels against your guidelines, and we review the work before it reaches you.
Task / Compare two answers
Example
Prompt
Summarise this support ticket in one sentence.
Answer A, preferred
The customer was charged twice and wants the second payment refunded.
Answer B
The customer has a question about a payment on their account.
Accurate
Follows the guideline
Note
B leaves out the double charge, the point of the ticket.
What we take on
Sold as finished work against your guideline, not as hours or seats. Text, voice and code, including specialist areas like finance.
Evaluation
Your model's outputs, rated against your rubric.
- Rating answers on a scale
- Comparing two answers
- Checking reasoning step by step
- Judging tone and naturalness
Validation
Your data checked, and fixed or flagged where it is wrong.
- Checking labels
- Checking transcripts against audio
- Checking annotations
- Spot checks before training
Languages and fields
The same work where general reviewers fall short.
- Any language
- Code
- Finance
- Voice and text
How the work is done
Checked first. Everyone on your project passed our check on that type of task before they see your data, and is picked for the language and field it needs.
Your guideline. We work to your instructions and examples, agree the bar with you in the pilot, and ask you when a case is left open instead of guessing.
Our review. We review the work ourselves before it is delivered. You deal with us, never with individuals.
Under NDA. From the first pilot, on data you choose to share.
What we don't do. Rent people by the hour, place them on your team, or take on a language we have not checked people for. We say so on the first call.
A typical pilot
- Scope
- One task type, in one language or field
- Size
- A few hundred items
- From you
- One call, the guideline, a sample, and someone to answer questions
- Terms
- Under NDA, on data you choose to share
- At the end
- The delivery, item by item, and a look at it together
What comes back
Each item with its rating, whether it follows your guideline, our review, and a note where something is wrong. In the format you work with.
Delivery / Batch 1
Example
| Item | Rating | Guideline | Review | Note |
|---|---|---|---|---|
| item_0141 | 4 / 5 | Followed | Approved | |
| item_0142 | 2 / 5 | Missed | Approved | Leaves out the refund |
| item_0143 | 5 / 5 | Followed | Approved | |
| item_0144 | 3 / 5 | Followed | Rating changed | Tone too casual |
Example, not a client's data
Send us your task
A line or two on what you want evaluated is enough to start. Mário Rodrigues, IdealJob's founder, reads every message and replies with the questions a pilot needs.
Prefer to talk? Book a call
What data teams ask us
- Who does the evaluation work?
- People who passed IdealJob's check on that type of task, picked for the language and field your task needs. They work on your project through IdealJob, our reviewers check what they hand in, and you deal with us, not with individuals.
- How do you check the quality?
- Everyone first passes a check on the task type. We work to your guideline, agree the bar with you in the pilot, and review the work ourselves before delivery. Each item comes back with the rating, whether it follows the guideline, our review and a note where something is wrong.
- Is our data kept confidential?
- The pilot runs under NDA, and only the people on your project and our reviewers see the data you share.
- Which languages and fields do you cover?
- Text, voice and code, in any language, and specialist fields like finance. We take on a language or field only when we have checked people for it, and tell you on the first call if we have not.
- What does a pilot ask of our team?
- One call, your guideline, a sample of the data, and someone who can answer questions while the pilot runs. A typical pilot is one task type and a few hundred items.
- Can you work in our annotation tool?
- Yes, if you can give our people and reviewers accounts. Otherwise we work from the files you send and return them in the same format.
- How is it priced?
- As finished work, not hours. We quote the pilot after the first call, once we know the task and its size.