code4thought

TRUSTWORTHY AI

SOFTWARE QUALITY

TRUSTWORTHY AI

Talk with Us
Talk with us

NYC AI Bias Audit

SOLUTIONS > NYC Bias Audit
The New York City Bias Audit Law (Local Law 144) regulates the use of automated employment decision tools (AEDT) for candidates and employees within New York city and requires that algorithm-based technologies for recruiting, hiring, or promotion be audited for bias before being used. In short, this law concerns you, if you are an employer or employment agency or software provider in New York City that builds or uses these tools.
The Department of Consumer and Worker Protection (DCWP) published its initial proposed rules in September 2022 and initially they would take effect on January 1, 2023. Due to concerns raised during the public comment period, including those raised by employers, employment agencies, law firms, AEDT developers and advocacy organizations, DCWP issued revised proposed rules and postponed enforcement for April, 15 2023. On April 6 2023, DCWP released final rules implementing Local Law 144 of 2021 and announced that enforcement will begin on July 5, 2023.

NYC AI Bias Audit Law Solution

With our AI Testing & Audit solution we help organizations ensure their AI-based systems can be trusted at every stage of their life cycle, whether they are purchased, built or just operated.
In order to specifically address the NYC AI Bias Law requirements, we have adapted our solution accordingly in order to provide:

Reliable AI
Βias Τesting

Our own proprietary platform PyThia, enables the analysis and testing of any type of data and AI models and provides Bias assessment per system, including, but not limited to:
  • Disparate impact analysis on persons (e.g., candidates, employees)
  • Per protected categories (e.g., gender, ethnicity, race)

Findings Report with Mitigation Measures

The results of our Bias Audit are presented to all relevant stakeholders accompanied with root cause analysis and mitigating measures, if necessary. Our fact-based reports will align your organization to address the root causes immediately and ensure compliance to legal requirements.

lorem ipsum lorem ipsum lorem ipsum

Summary of Results for publishing

Our findings from the bias audit report are collated into a summary of results, including the selection rates and impact ratios of protected categories, to be shared on your company website, as required by the NYC regulations.

lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsumlorem ipsumlorem ipsum

Why us?

Independent, objective testing & advisory
Model and Data-Agnostic solution
Data-driven analysis of PyThia
Expertise in comprehensive AI testing & auditing
Root cause analysis & Mitigation Measures
Structured auditing processes for Bias Testing

NYC Local Law 144 Summary

Local Law 144 of 2021 prohibits employers and employment agencies from using an automated employment decision tool (AEDT) to screen a candidate or employee for an employment decision unless:
– the tool has been subject to a bias audit by an independent auditor within one year of the use of the tool,
– information about the bias audit is publicly available,
– certain notices have been provided to employees or job candidates.

Bias audit

“Bias audit” refers to an impartial evaluation by an independent auditor. Such bias audit shall include but not be limited to the testing whether your system is discriminating against any protected categories, such as gender, sex, ethnicity, or race.
Specific metrics as a minimum have been defined and include:
– selection rate or scoring rate for each category
– impact ratio for each category
– intersectional analysis of disparate impact
The bias audit is necessary even if the employer is not using the AEDT to make the final hiring decision and is using it at a prior stage, for example to screen at an early point in the application process or to score applicants for “culture fit’.

Published Results

Prior to the use of an AEDT, an employer or employment agency in the city must make the following publicly available on the employment section of their website:
(1) The date of the most recent bias audit of the AEDT and a summary of the results. Summary of results must include:
(a) the source and explanation of the data used to conduct the bias audit
(b) the number of individuals the AEDT assessed that fall within an unknown category
(c) the number of applicants or candidates
(d) the selection or scoring rates, as applicable, and the impact ratios for all categories
(2) The distribution date of the AEDT (the date the employer or employment agency began using a specific AEDT)
The summary of results and distribution date must be kept posted for at least 6 months after last using the AEDT for an employment decision.

Notice to Candidates

An employer or employment agency must provide notice concerning the use of the AEDT to a candidate for employment who resides in the city at least 10 business days prior to use. Notice can be any one of the following:
(1) on the employment section of its website
(2) in a job posting
(3) via U.S. mail or e-mail
(4) in a written policy or procedure, in case of promotion.
Moreover, an employer or employment agency must:
(1) Provide information on the employment section of its website about its AEDT data retention policy, the type of data collected for the AEDT, and the source of the data;
(2) Post instructions on the employment section of its website for how to make a written request for such information. If a written request is received, he has to provide such information within 30 days.

Penalties for non-compliance

$500 for a first violation and each additional violation occurring on the same day. $500 – $1,500 for each subsequent violation.

Frequently Asked Questions

Who needs to comply to the NYC Bias Audit Law?
All employers and employment agencies who meet the criteria below must conduct or comply to a bias audit by April 15th, 2023:
  • They use an automated employment decision tool (e.g., resume screening) whose output such as a score, classification or recommendation is used
  • To evaluate candidates or employees
  • Seeking a position or promotion
  • And are residing in New York City (this also includes remote work positions).
Suppliers of the automated employment decision tool may be asked by their employer clients (be it employer organizations or employment agencies) to provide the audit as a condition of purchase or to work with the auditor the employer client selects.
What is an Automated Employment Decision Tool?
The term “automated employment decision tool” means any computational process, derived from machine learning, statistical modeling, data analytics, or artificial intelligence, that issues simplified output, including a score, classification, or recommendation, that is used to substantially assist or replace discretionary decision making for making employment decisions that impact natural persons. “To substantially assist or replace discretionary decision making” means:
  1. to rely solely on a simplified output (score, tag, classification, ranking, etc.), with no other factors considered; or
  2. to use a simplified output as one of a set of criteria where the simplified output is weighted more than any other criterion in the set; or
  3. to use a simplified output to overrule conclusions derived from other factors including human decision-making.
“Machine learning, statistical modeling, data analytics, or artificial intelligence” means a group of mathematical, computer based techniques:
  1. that generate a prediction, meaning an expected outcome for an observation, such as an assessment of a candidate’s fit or likelihood of success, or that generate a classification, meaning an assignment of an observation to a group, such as categorizations based on skill sets or aptitude; or
  2. for which a computer at least in part identifies the inputs, the relative importance placed on those inputs, and, if applicable, other parameters for the models in order to improve the accuracy of the prediction or classification.
Simplified output. “Simplified output” means a prediction or classification as specified in the definition for “machine learning, statistical modelling, data analytics, or artificial intelligence.” A simplified output may take the form of a score (e.g., rating a candidate’s estimated technical skills), tag or categorization (e.g., categorizing a candidate’s resume based on key words, assigning a skill or trait to a candidate), recommendation (e.g., whether a candidate should be given an interview), or ranking (e.g., arranging a list of candidates based on how well their cover letters match the job description). It does not refer to the output from analytical tools that translate or transcribe existing text, e.g., convert a resume from a PDF or transcribe a video or audio interview.
What are some examples of an Automated Employment Decision Tool?
Examples of AEDTs include tools that screen resumes, recommend whether a candidate should be given an interview, as well as those that score applicants for a “culture fit” or other assessments, such as game-based, image based or psychometric tools. Other examples include rating a candidate’s estimated technical skills, categorizing a candidate’s resume based on key words, assigning a skill or trait to a candidate, arranging a list of candidates based on how well their cover letters match the job description and others.
What is a candidate for employment?
“Candidate for employment” means a person who has applied for a specific employment position by submitting the necessary information or items in the format required by the employer or employment agency.
How is screening defined?
“Screen” means to make a determination about whether a candidate for employment or employee being considered for promotion should be selected or advanced in the hiring or promotion process.
What does “employment decision” refer to according to NYC Local Law 144?
The term “employment decision” means to screen candidates for employment or employees for promotion within the city.
What is Selection Rate?
“Selection rate” is calculated by dividing the number of individuals in the category moving forward or assigned a classification by the total number of individuals in the category who applied for a position or were considered for promotion. Example
What is Scoring Rate?
“Scoring Rate” means the rate at which individuals in a category receive a score above the sample’s median score, where the score has been calculated by an AEDT.
How is Impact Ratio calculated?
“Impact ratio” means the selection or scoring rate for a category divided by the selection rate of the most selected or highest scoring category respectively

Example

What does Intersectional Analysis involve?

Selection/Scoring rate and Impact Ratio must separately calculate the impact of the AEDT on:

  1. Sex categories
    i.e., impact ratio for selection of male candidates vs female candidates,
  2. Race/Ethnicity categories
    e.g., impact ratio for selection of Hispanic or Latino candidates vs Black or African American [Not Hispanic or Latino] candidates
  3. intersectional categories of sex, ethnicity, and race
    e.g., impact ratio for selection of Hispanic or Latino male candidates vs. Not Hispanic or Latino Black or African American female candidates.

Example

Τhe number of individuals the AEDT assessed that are not included in the calculations because they fall within an unknown category,  should be mentioned in a respective note in a the summary of results.

Example Note: The AEDT was also used to assess 250 individuals with an unknown sex or race/ethnicity category. Data on those individuals was not included in the calculations above.

What are the requirements for an Independent Auditor?
“Independent auditor” means a person or group that is capable of exercising objective and impartial judgment on all issues within the scope of a bias audit of an AEDT. In order an auditor to be considered independent, he must:
  1. not be involved in using, developing, or distributing the AEDT
  2. not have an employment relationship with the employer or the employment agency or the AEDT software vendor at any point during the bias audit or
  3. have no financial interest in the employer or the employment agency or the AEDT software vendor at any point during the bias audit
What are the Data Requirements for the Bias Audit?
A bias audit conducted must use historical data of the AEDT. If insufficient historical data is available to conduct a statistically significant bias audit, test data may be used instead. However, if a bias audit uses test data, the summary of results of the bias audit must explain why historical data was not used and describe how the test data used was generated and obtained. A bias audit of an AEDT used by multiple employers or employment agencies may use the historical data of any employers or employment agencies that use the AEDT. However, an employer or employment agency may rely on a bias audit of an AEDT that uses the historical data of other employers or employment agencies only if it has also provided its own historical data from AEDT use to the auditor for the bias audit or if it has never used the AEDT.
What is the standard process and duration of an AI Bias Audit project?
As a first step, a kick-off session with the Client takes place, followed by technical interviews with Client’s team. Based on the information and data gathered, code4thought team proceeds with the AI system testing and respective technical analysis, which is presented and validated with the client in a respective session.
This Phase (Analysis Phase) usually takes 1-3 weeks depending on the Project.
The Reporting Phase follows, during which code4thought prepares the results, which are presented to the Client for validation and finally we have the final report session.
Reporting Phase usually takes 1-3 weeks, as well.
  •    WE' D LOVE TO HELP YOU  
  •    WE' D LOVE TO HELP YOU  
  •    WE' D LOVE TO HELP YOU  
  •   WE' D LOVE TO HELP YOU  
  •    WE' D LOVE TO HELP YOU  
  •    WE' D LOVE TO HELP YOU  

Let's get started with your AI Bias Audit!

    Let's get started with your
    AI Bias Audit!

      FURTHER READING