For teams building AI

Human evaluation for your models.

IdealJob takes on LLM evaluation and data validation for teams building AI. People we checked on the task type rate your model's outputs and check your labels against your guidelines, and we review the work before it reaches you.

Task / Compare two answers

Example

Prompt

Summarise this support ticket in one sentence.

Answer A, preferred

The customer was charged twice and wants the second payment refunded.

Answer B

The customer has a question about a payment on their account.

Accurate

1 2 3 4 5

Follows the guideline

Yes No

Note

B leaves out the double charge, the point of the ticket.

Second review Approved

What we take on

Sold as finished work against your guideline, not as hours or seats. Text, voice and code, including specialist areas like finance.

Evaluation

Your model's outputs, rated against your rubric.

  • Rating answers on a scale
  • Comparing two answers
  • Checking reasoning step by step
  • Judging tone and naturalness

Validation

Your data checked, and fixed or flagged where it is wrong.

  • Checking labels
  • Checking transcripts against audio
  • Checking annotations
  • Spot checks before training

Languages and fields

The same work where general reviewers fall short.

  • Any language
  • Code
  • Finance
  • Voice and text

How the work is done

Checked first. Everyone on your project passed our check on that type of task before they see your data, and is picked for the language and field it needs.

Your guideline. We work to your instructions and examples, agree the bar with you in the pilot, and ask you when a case is left open instead of guessing.

Our review. We review the work ourselves before it is delivered. You deal with us, never with individuals.

Under NDA. From the first pilot, on data you choose to share.

What we don't do. Rent people by the hour, place them on your team, or take on a language we have not checked people for. We say so on the first call.

A typical pilot

Scope
One task type, in one language or field
Size
A few hundred items
From you
One call, the guideline, a sample, and someone to answer questions
Terms
Under NDA, on data you choose to share
At the end
The delivery, item by item, and a look at it together

What comes back

Each item with its rating, whether it follows your guideline, our review, and a note where something is wrong. In the format you work with.

Delivery / Batch 1

Example

Item Rating Guideline Review Note
item_0141 4 / 5 Followed Approved
item_0142 2 / 5 Missed Approved Leaves out the refund
item_0143 5 / 5 Followed Approved
item_0144 3 / 5 Followed Rating changed Tone too casual

Example, not a client's data

Send us your task

A line or two on what you want evaluated is enough to start. Mário Rodrigues, IdealJob's founder, reads every message and replies with the questions a pilot needs.

Prefer to talk? Book a call

We reply by email.

How we handle your details: privacy policy

What data teams ask us

Who does the evaluation work?
People who passed IdealJob's check on that type of task, picked for the language and field your task needs. They work on your project through IdealJob, our reviewers check what they hand in, and you deal with us, not with individuals.
How do you check the quality?
Everyone first passes a check on the task type. We work to your guideline, agree the bar with you in the pilot, and review the work ourselves before delivery. Each item comes back with the rating, whether it follows the guideline, our review and a note where something is wrong.
Is our data kept confidential?
The pilot runs under NDA, and only the people on your project and our reviewers see the data you share.
Which languages and fields do you cover?
Text, voice and code, in any language, and specialist fields like finance. We take on a language or field only when we have checked people for it, and tell you on the first call if we have not.
What does a pilot ask of our team?
One call, your guideline, a sample of the data, and someone who can answer questions while the pilot runs. A typical pilot is one task type and a few hundred items.
Can you work in our annotation tool?
Yes, if you can give our people and reviewers accounts. Otherwise we work from the files you send and return them in the same format.
How is it priced?
As finished work, not hours. We quote the pilot after the first call, once we know the task and its size.