European Parliament 2019-2024 {CULT}Committee on Culture and Education 2021/0106 (COD) {21/02/2022}21.2.2022
of the Committee on Culture and Education for the Committee on the Internal Market and Consumer Protection and the Committee on Civil Liberties, Justice and Home Affairs on the proposal for a regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union Legislative Acts
Rapporteur for opinion (*): Marcel Kolaja (*) Associated committee – Rule 57 of the Rules of Procedure PA_Legam
On 24 April 2021, the European Commission published its legislative proposal laying down harmonised rules on artificial intelligence (AI Act), which introduces a regulatory framework with the objective of ensuring that AI systems placed on the European Union market are safe to use and respect fundamental rights and European Union values.
Furthermore, the proposal facilitates development of a single market for lawful, safe and trustworthy AI applications, enhances governance and effective enforcement of existing law on fundamental rights and safety requirements applicable to AI systems and ensures legal certainty to facilitate investment and innovation in AI. Overall, the Rapporteur welcomes the European Commission’s proposal; however, would like to suggest a few amendments mainly to extend the list of high-risk AI applications in areas of education, media and culture under Annex III and to modify certain provisions related to banned practices under Article 5. More specifically, the Rapporteur reflects on the increased deployment of AI technologies in education and training facilities.
Therefore, he proposes listing, among high-risk technologies, also those AI technologies used for monitoring of students during tests and technologies used to determine an area or a programme a student should study. Regarding media and culture, the Rapporteur suggests listing high-risk AI technologies used to create or disseminate machine-generated news articles used by news media outlets and AI technologies used to recommend or rank audiovisual content. In addition, the Rapporteur proposes to extend the ban on deployment of social scoring systems to usage by public and private entities given the inherent threat of discrimination and exclusion of certain groups or individuals.
Finally, in light of the danger that deployment of remote biometric identification systems in publicly accessible places poses to citizens’ fundamental rights, freedom of assembly, work of investigative journalists, activists and political representatives, the Rapporteur proposes to ban deployment of such technologies in publicly accessible places following up on the Parliament resolution of 6 October 2021 on “Artificial intelligence in criminal law and its use by the police and judicial authorities in criminal matters”.
The Committee on Culture and Education calls on the Committee on the Internal Market and Consumer Protection and the Committee on Civil Liberties, Justice and Home Affairs, as the committees responsible, to take into account the following amendments: <RepeatBlock-Amend> Amendment 1 Proposal for a regulation Recital 1 Text proposed by the Commission Amendment
(1) The purpose of this Regulation is to improve the functioning of the internal market by laying down a uniform legal framework in particular for the development, marketing and use of artificial intelligence in conformity with Union values.
This Regulation pursues a number of overriding reasons of public interest, such as a high level of protection of health, safety and fundamental rights, and it ensures the free movement of AI-based goods and services cross-border, thus preventing Member States from imposing restrictions on the development, marketing and use of AI systems, unless explicitly authorised by this Regulation. (1) The purpose of this Regulation is to improve the functioning of the internal market by laying down a uniform legal framework in particular for the development, marketing and use of artificial intelligence in conformity with Union values.
This Regulation pursues a number of overriding reasons of public interest, such as a high level of protection of health, safety and fundamental rights, as well as of society and the environment, and it ensures the free movement of AI-based goods and services cross-border, thus preventing Member States from imposing restrictions on the development, marketing and use of AI systems, unless explicitly authorised by this Regulation. Or. {EN}en Amendment 2 Proposal for a regulation Recital 4
Text proposed by the Commission Amendment
(4) At the same time, depending on the circumstances regarding its specific application and use, artificial intelligence may generate risks and cause harm to public interests and rights that are protected by Union law. Such harm might be material or immaterial.
(4) At the same time, depending on the circumstances regarding its specific application and use, artificial intelligence may generate risks and cause harm to public interests and rights that are protected by Union law, whether individual, societal or environmental. Such harm might be material or immaterial. Or. {EN}en Amendment 3 Proposal for a regulation Recital 5
Text proposed by the Commission Amendment
(5) A Union legal framework laying down harmonised rules on artificial intelligence is therefore needed to foster the development, use and uptake of artificial intelligence in the internal market that at the same time meets a high level of protection of public interests, such as health and safety and the protection of fundamental rights, as recognised and protected by Union law. To achieve that objective, rules regulating the placing on the market and putting into service of certain AI systems should be laid down, thus ensuring the smooth functioning of the internal market and allowing those systems to benefit from the principle of free movement of goods and services.
By laying down those rules, this Regulation supports the objective of the Union of being a global leader in the development of secure, trustworthy and ethical artificial intelligence, as stated by the European Council33 , and it ensures the protection of ethical principles, as specifically requested by the European Parliament34 . (5) A Union legal framework laying down harmonised rules on artificial intelligence is therefore needed to foster the development, use and uptake of artificial intelligence in the internal market that at the same time meets a high level of protection of public interests, such as health and safety and the protection of fundamental rights, as recognised and protected by Union law, as well as of society and the environment.
To achieve that objective, rules regulating the placing on the market and putting into service of certain AI systems should be laid down, thus ensuring the smooth functioning of the internal market and allowing those systems to benefit from the principle of free movement of goods and services. By laying down those rules, this Regulation supports the objective of the Union of being a global leader in the development of secure, trustworthy and ethical artificial intelligence, as stated by the European Council33 , and it ensures the protection of ethical principles, as specifically requested by the European Parliament34 .
33 European Council, Special meeting of the European Council (1 and 2 October 2020) – Conclusions, EUCO 13/20, 2020, p. 6. 33 European Council, Special meeting of the European Council (1 and 2 October 2020) – Conclusions, EUCO 13/20, 2020, p. 6. 34 European Parliament resolution of 20 October 2020 with recommendations to the Commission on a framework of ethical aspects of artificial intelligence, robotics and related technologies, 2020/2012(INL). 34 European Parliament resolution of 20 October 2020 with recommendations to the Commission on a framework of ethical aspects of artificial intelligence, robotics and related technologies, 2020/2012(INL). Or.
{EN}en Amendment 4 Proposal for a regulation Recital 8 Text proposed by the Commission Amendment
(8) The notion of remote biometric identification system as used in this Regulation should be defined functionally, as an AI system intended for the identification of natural persons at a distance through the comparison of a person’s biometric data with the biometric data contained in a reference database, and without prior knowledge whether the targeted person will be present and can be identified, irrespectively of the particular technology, processes or types of biometric data used.
Considering their different characteristics and manners in which they are used, as well as the different risks involved, a distinction should be made between ‘real-time’ and ‘post’ remote biometric identification systems. In the case of ‘real-time’ systems, the capturing of the biometric data, the comparison and the identification occur all instantaneously, near-instantaneously or in any event without a significant delay. In this regard, there should be no scope for circumventing the rules of this Regulation on the ‘real-time’ use of the AI systems in question by providing for minor delays. ‘Real-time’ systems involve the use of ‘live’ or ‘near-‘live’ material, such as video footage, generated by a camera or other device with similar functionality.
In the case of ‘post’ systems, in contrast, the biometric data have already been captured and the comparison and identification occur only after a significant delay. This involves material, such as pictures or video footage generated by closed circuit television cameras or private devices, which has been generated before the use of the system in respect of the natural persons concerned. (8) The notion of remote biometric identification system as used in this Regulation should be defined functionally, as an AI system intended for the identification of natural persons at a distance through the comparison of a person’s biometric data with the biometric data contained in a reference database, irrespectively of the particular technology, processes or types of biometric data used.
Considering their different characteristics and manners in which they are used, as well as the different risks involved, a distinction should be made between ‘real-time’ and ‘post’ remote biometric identification systems. In the case of ‘real-time’ systems, the capturing of the biometric data, the comparison and the identification occur all instantaneously, near-instantaneously or in any event without a significant delay. In this regard, there should be no scope for circumventing the rules of this Regulation on the ‘real-time’ use of the AI systems in question by providing for minor delays. ‘Real-time’ systems involve the use of ‘live’ or ‘near-‘live’ material, such as video footage, generated by a camera or other device with similar functionality.
In the case of ‘post’ systems, in contrast, the biometric data have already been captured and the comparison and identification occur only after a significant delay. This involves material, such as pictures or video footage generated by closed circuit television cameras or private devices, which has been generated before the use of the system in respect of the natural persons concerned. Or. {EN}en Amendment 5 Proposal for a regulation Recital 9
Text proposed by the Commission Amendment
(9) For the purposes of this Regulation the notion of publicly accessible space should be understood as referring to any physical place that is accessible to the public, irrespective of whether the place in question is privately or publicly owned. Therefore, the notion does not cover places that are private in nature and normally not freely accessible for third parties, including law enforcement authorities, unless those parties have been specifically invited or authorised, such as homes, private clubs, offices, warehouses and factories. Online spaces are not covered either, as they are not physical spaces.
However, the mere fact that certain conditions for accessing a particular space may apply, such as admission tickets or age restrictions, does not mean that the space is not publicly accessible within the meaning of this Regulation. Consequently, in addition to public spaces such as streets, relevant parts of government buildings and most transport infrastructure, spaces such as cinemas, theatres, shops and shopping centres are normally also publicly accessible. Whether a given space is accessible to the public should however be determined on a case-by-case basis, having regard to the specificities of the individual situation at hand.
(9) For the purposes of this Regulation the notion of publicly accessible space should be understood as referring to any physical or virtual place that is accessible to the public, irrespective of whether the place in question is privately or publicly owned. Therefore, the notion does not cover places that are private in nature and normally not freely accessible for third parties, including law enforcement authorities, unless those parties have been specifically invited or authorised, such as homes, private clubs, offices, warehouses and factories. The same principle should apply to virtual publicly accessible spaces.
However, the mere fact that certain conditions for accessing a particular space may apply, such as admission tickets or age restrictions, does not mean that the space is not publicly accessible within the meaning of this Regulation. Consequently, in addition to public spaces such as streets, relevant parts of government buildings and most transport infrastructure, spaces such as cinemas, theatres, shops and shopping centres are normally also publicly accessible. Whether a given space is accessible to the public should however be determined on a case-by-case basis, having regard to the specificities of the individual situation at hand. Or.
{EN}en Amendment 6 Proposal for a regulation Recital 13
Text proposed by the Commission Amendment
(13) In order to ensure a consistent and high level of protection of public interests as regards health, safety and fundamental rights, common normative standards for all high-risk AI systems should be established. Those standards should be consistent with the Charter of fundamental rights of the European Union (the Charter) and should be non-discriminatory and in line with the Union’s international trade commitments.
(13) In order to ensure a consistent and high level of protection of public interests as regards health, safety and fundamental rights, as well as of society and the environment, common normative standards for all high-risk AI systems should be established. Those standards should be consistent with the Charter of fundamental rights of the European Union (the Charter) and should be non-discriminatory and in line with the Union’s international trade commitments. Or. {EN}en Amendment 7 Proposal for a regulation Recital 16
Text proposed by the Commission Amendment
(16) The placing on the market, putting into service or use of certain AI systems intended to distort human behaviour, whereby physical or psychological harms are likely to occur, should be forbidden. Such AI systems deploy subliminal components individuals cannot perceive or exploit vulnerabilities of children and people due to their age, physical or mental incapacities. They do so with the intention to materially distort the behaviour of a person and in a manner that causes or is likely to cause harm to that or another person. The intention may not be presumed if the distortion of human behaviour results from factors external to the AI system which are outside of the control of the provider or the user.
Research for legitimate purposes in relation to such AI systems should not be stifled by the prohibition, if such research does not amount to use of the AI system in human-machine relations that exposes natural persons to harm and such research is carried out in accordance with recognised ethical standards for scientific research. (16) The placing on the market, putting into service or use of certain AI systems intended to distort human behaviour, whereby material or non-material harm, including physical, psychological or economic harm is likely to occur, should be forbidden. Such AI systems deploy subliminal components individuals cannot perceive or exploit vulnerabilities of children and people due to their age, physical or mental capacity.
They do so with the intention to materially distort the behaviour of a person and in a manner that causes or is likely to cause harm to that or another person. The intention may not be presumed if the distortion of human behaviour results from factors external to the AI system which are outside of the control of the provider or the user. Research for legitimate purposes in relation to such AI systems should not be stifled by the prohibition, if such research does not amount to use of the AI system in human-machine relations that exposes natural persons to harm and such research is carried out in accordance with recognised ethical standards for scientific research. Or.
{EN}en Amendment 8 Proposal for a regulation Recital 17 Text proposed by the Commission Amendment
(17) AI systems providing social scoring of natural persons for general purpose by public authorities or on their behalf may lead to discriminatory outcomes and the exclusion of certain groups. They may violate the right to dignity and non-discrimination and the values of equality and justice. Such AI systems evaluate or classify the trustworthiness of natural persons based on their social behaviour in multiple contexts or known or predicted personal or personality characteristics.
The social score obtained from such AI systems may lead to the detrimental or unfavourable treatment of natural persons or whole groups thereof in social contexts, which are unrelated to the context in which the data was originally generated or collected or to a detrimental treatment that is disproportionate or unjustified to the gravity of their social behaviour. Such AI systems should be therefore prohibited. (17) AI systems providing social scoring of natural persons for general purpose may lead to discriminatory outcomes and the exclusion of certain groups. They may violate the right to dignity and non-discrimination and the values of equality and justice.
Such AI systems evaluate or classify the trustworthiness of natural persons based on their social behaviour in multiple contexts or known or predicted personal or personality characteristics. Such AI systems should be therefore prohibited. Or. {EN}en Amendment 9 Proposal for a regulation Recital 18
Text proposed by the Commission Amendment
(18) The use of AI systems for ‘real-time’ remote biometric identification of natural persons in publicly accessible spaces for the purpose of law enforcement is considered particularly intrusive in the rights and freedoms of the concerned persons, to the extent that it may affect the private life of a large part of the population, evoke a feeling of constant surveillance and indirectly dissuade the exercise of the freedom of assembly and other fundamental rights. In addition, the immediacy of the impact and the limited opportunities for further checks or corrections in relation to the use of such systems operating in ‘real-time’ carry heightened risks for the rights and freedoms of the persons that are concerned by law enforcement activities.
(18) The use of AI systems for ‘real-time’ and ‘post’ remote biometric identification of natural persons in publicly accessible spaces is considered particularly intrusive in the rights and freedoms of the concerned persons, to the extent that it may affect the private life of a large part of the population, evoke a feeling of constant surveillance and indirectly dissuade the exercise of the freedom of assembly and other fundamental rights. In addition, the immediacy of the impact and the limited opportunities for further checks or corrections in relation to the use of such systems operating in ‘real-time’ carry heightened risks for the rights and freedoms of the persons. Or.
{EN}en Amendment 10 Proposal for a regulation Recital 19 Text proposed by the Commission Amendment
(19) The use of those systems for the purpose of law enforcement should therefore be prohibited, except in three exhaustively listed and narrowly defined situations, where the use is strictly necessary to achieve a substantial public interest, the importance of which outweighs the risks.
Those situations involve the search for potential victims of crime, including missing children; certain threats to the life or physical safety of natural persons or of a terrorist attack; and the detection, localisation, identification or prosecution of perpetrators or suspects of the criminal offences referred to in Council Framework Decision 2002/584/JHA38 if those criminal offences are punishable in the Member State concerned by a custodial sentence or a detention order for a maximum period of at least three years and as they are defined in the law of that Member State.
Such threshold for the custodial sentence or detention order in accordance with national law contributes to ensure that the offence should be serious enough to potentially justify the use of ‘real-time’ remote biometric identification systems.
Moreover, of the 32 criminal offences listed in the Council Framework Decision 2002/584/JHA, some are in practice likely to be more relevant than others, in that the recourse to ‘real-time’ remote biometric identification will foreseeably be necessary and proportionate to highly varying degrees for the practical pursuit of the detection, localisation, identification or prosecution of a perpetrator or suspect of the different criminal offences listed and having regard to the likely differences in the seriousness, probability and scale of the harm or possible negative consequences. (19) Given the detrimental effect of those systems to freedoms a liberties of impacted individuals, the use of those systems in publicly accessible spaces should be prohibited.
38 Council Framework Decision 2002/584/JHA of 13 June 2002 on the European arrest warrant and the surrender procedures between Member States (OJ L 190, 18.7.2002, p. 1). Or. {EN}en Amendment 11 Proposal for a regulation Recital 20
Text proposed by the Commission Amendment
(20) In order to ensure that those systems are used in a responsible and proportionate manner, it is also important to establish that, in each of those three exhaustively listed and narrowly defined situations, certain elements should be taken into account, in particular as regards the nature of the situation giving rise to the request and the consequences of the use for the rights and freedoms of all persons concerned and the safeguards and conditions provided for with the use. In addition, the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement should be subject to appropriate limits in time and space, having regard in particular to the evidence or indications regarding the threats, the victims or perpetrator. The reference database of persons should be appropriate for each use case in each of the three situations mentioned above. deleted Or. {EN}en Amendment 12 Proposal for a regulation Recital 21
Text proposed by the Commission Amendment
(21) Each use of a ‘real-time’ remote biometric identification system in publicly accessible spaces for the purpose of law enforcement should be subject to an express and specific authorisation by a judicial authority or by an independent administrative authority of a Member State. Such authorisation should in principle be obtained prior to the use, except in duly justified situations of urgency, that is, situations where the need to use the systems in question is such as to make it effectively and objectively impossible to obtain an authorisation before commencing the use. In such situations of urgency, the use should be restricted to the absolute minimum necessary and be subject to appropriate safeguards and conditions, as determined in national law and specified in the context of each individual urgent use case by the law enforcement authority itself. In addition, the law enforcement authority should in such situations seek to obtain an authorisation as soon as possible, whilst providing the reasons for not having been able to request it earlier. deleted Or. {EN}en Amendment 13 Proposal for a regulation Recital 22
Text proposed by the Commission Amendment
(22) Furthermore, it is appropriate to provide, within the exhaustive framework set by this Regulation that such use in the territory of a Member State in accordance with this Regulation should only be possible where and in as far as the Member State in question has decided to expressly provide for the possibility to authorise such use in its detailed rules of national law. Consequently, Member States remain free under this Regulation not to provide for such a possibility at all or to only provide for such a possibility in respect of some of the objectives capable of justifying authorised use identified in this Regulation. deleted Or. {EN}en Amendment 14 Proposal for a regulation Recital 23
Text proposed by the Commission Amendment
(23) The use of AI systems for ‘real-time’ remote biometric identification of natural persons in publicly accessible spaces for the purpose of law enforcement necessarily involves the processing of biometric data. The rules of this Regulation that prohibit, subject to certain exceptions, such use, which are based on Article 16 TFEU, should apply as lex specialis in respect of the rules on the processing of biometric data contained in Article 10 of Directive (EU) 2016/680, thus regulating such use and the processing of biometric data involved in an exhaustive manner.
Therefore, such use and processing should only be possible in as far as it is compatible with the framework set by this Regulation, without there being scope, outside that framework, for the competent authorities, where they act for purpose of law enforcement, to use such systems and process such data in connection thereto on the grounds listed in Article 10 of Directive (EU) 2016/680. In this context, this Regulation is not intended to provide the legal basis for the processing of personal data under Article 8 of Directive 2016/680.
However, the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for purposes other than law enforcement, including by competent authorities, should not be covered by the specific framework regarding such use for the purpose of law enforcement set by this Regulation. Such use for purposes other than law enforcement should therefore not be subject to the requirement of an authorisation under this Regulation and the applicable detailed rules of national law that may give effect to it. (23) The use of AI systems for ‘real-time’ and ‘post’ remote biometric identification of natural persons in publicly accessible spaces necessarily involves the processing of biometric data.
The rules of this Regulation that prohibit such use, which are based on Article 16 TFEU, should apply as lex specialis in respect of the rules on the processing of biometric data contained in Article 10 of Directive (EU) 2016/680, thus regulating such use and the processing of biometric data involved in an exhaustive manner. Or. {EN}en Amendment 15 Proposal for a regulation Recital 24
Text proposed by the Commission Amendment
(24) Any processing of biometric data and other personal data involved in the use of AI systems for biometric identification, other than in connection to the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement as regulated by this Regulation, including where those systems are used by competent authorities in publicly accessible spaces for other purposes than law enforcement, should continue to comply with all requirements resulting from Article 9(1) of Regulation (EU) 2016/679, Article 10(1) of Regulation (EU) 2018/1725 and Article 10 of Directive (EU) 2016/680, as applicable. deleted Or. {EN}en Amendment 16 Proposal for a regulation Recital 27
Text proposed by the Commission Amendment
(27) High-risk AI systems should only be placed on the Union market or put into service if they comply with certain mandatory requirements. Those requirements should ensure that high-risk AI systems available in the Union or whose output is otherwise used in the Union do not pose unacceptable risks to important Union public interests as recognised and protected by Union law. AI systems identified as high-risk should be limited to those that have a significant harmful impact on the health, safety and fundamental rights of persons in the Union and such limitation minimises any potential restriction to international trade, if any.
(27) High-risk AI systems should only be placed on the Union market or put into service if they comply with certain mandatory requirements. Those requirements should ensure that high-risk AI systems available in the Union or whose output is otherwise used in the Union do not pose unacceptable risks to important Union public interests as recognised and protected by Union law. AI systems identified as high-risk should be limited to those that have a significant harmful impact on the health, safety and fundamental rights of persons in the Union, as well as on society and on the environment, and such limitation minimises any potential restriction to international trade, if any. Or.
{EN}en Amendment 17 Proposal for a regulation Recital 33 Text proposed by the Commission Amendment
(33) Technical inaccuracies of AI systems intended for the remote biometric identification of natural persons can lead to biased results and entail discriminatory effects. This is particularly relevant when it comes to age, ethnicity, sex or disabilities. Therefore, ‘real-time’ and ‘post’ remote biometric identification systems should be classified as high-risk. In view of the risks that they pose, both types of remote biometric identification systems should be subject to specific requirements on logging capabilities and human oversight.
(33) Technical inaccuracies of AI systems intended for the biometric identification of natural persons can lead to biased results and entail discriminatory effects. This is particularly relevant when it comes to age, ethnicity, sex or disabilities. Therefore, ‘real-time’ and ‘post’ remote biometric identification systems, as well as non-remote biometric identification systems intended to be used in publicly accessible spaces, workplaces and education and training institutions should be classified as high-risk. In view of the risks that they pose, both types of biometric identification systems should be subject to specific requirements on logging capabilities and human oversight. Or.
{EN}en Amendment 18 Proposal for a regulation Recital 35 Text proposed by the Commission Amendment
(35) AI systems used in education or vocational training, notably for determining access or assigning persons to educational and vocational training institutions or to evaluate persons on tests as part of or as a precondition for their education should be considered high-risk, since they may determine the educational and professional course of a person’s life and therefore affect their ability to secure their livelihood.
When improperly designed and used, such systems may violate the right to education and training as well as the right not to be discriminated against and perpetuate historical patterns of discrimination. (35) AI systems used in education or training, notably for determining access or assigning persons to educational and training institutions to evaluate persons on tests as part of or as a precondition for their education or for determining the areas of study a student should follow should be considered high-risk, since they may determine the educational and professional course of a person’s life and therefore affect their ability to secure their livelihood.
When improperly designed and used, such systems may violate the right to education and training as well as the right not to be discriminated against and perpetuate historical patterns of discrimination. AI systems used to monitor students during tests at education and training institutions should be considered high-risk, since they are also interfering with students’ rights to privacy and data protection. Or. {EN}en Amendment 19 Proposal for a regulation Recital 35 a (new) Text proposed by the Commission Amendment
(35a) AI systems used in media and culture, in particular those that create and disseminate machine-generated news articles and those that suggest or prioritize audiovisual content should be considered high-risk, since those systems may influence society, spread disinformation and misinformation, have a negative impact on elections and other democratic processes and impact cultural and linguistic diversity.
Or. {EN}en Amendment 20 Proposal for a regulation Recital 36
Text proposed by the Commission Amendment
(36) AI systems used in employment, workers management and access to self-employment, notably for the recruitment and selection of persons, for making decisions on promotion and termination and for task allocation, monitoring or evaluation of persons in work-related contractual relationships, should also be classified as high-risk, since those systems may appreciably impact future career prospects and livelihoods of these persons. Relevant work-related contractual relationships should involve employees and persons providing services through platforms as referred to in the Commission Work Programme 2021. Such persons should in principle not be considered users within the meaning of this Regulation.
Throughout the recruitment process and in the evaluation, promotion, or retention of persons in work-related contractual relationships, such systems may perpetuate historical patterns of discrimination, for example against women, certain age groups, persons with disabilities, or persons of certain racial or ethnic origins or sexual orientation. AI systems used to monitor the performance and behaviour of these persons may also impact their rights to data protection and privacy.
(36) AI systems used in employment, workers management and access to self-employment, notably for the recruitment and selection of persons, for making decisions on promotion and termination and for task allocation, for monitoring compliance with workplace rules and for monitoring or evaluation of persons in work-related relationships, should also be classified as high-risk, since those systems may appreciably impact future career prospects and livelihoods of these persons. Relevant work-related relationships should involve employees and persons providing services through platforms as referred to in the Commission Work Programme 2021. Such persons should in principle not be considered users within the meaning of this Regulation.
Throughout the recruitment process and in the evaluation, promotion, or retention of persons in work-related relationships, such systems may perpetuate historical patterns of discrimination, for example against women, certain age groups, persons with disabilities, or persons of certain racial or ethnic origins or sexual orientation. AI systems used to monitor the performance and behaviour of these persons may also impact their rights to data protection and privacy. Or. {EN}en Amendment 21 Proposal for a regulation Recital 85
Text proposed by the Commission Amendment
(85) In order to ensure that the regulatory framework can be adapted where necessary, the power to adopt acts in accordance with Article 290 TFEU should be delegated to the Commission to amend the techniques and approaches referred to in Annex I to define AI systems, the Union harmonisation legislation listed in Annex II, the high-risk AI systems listed in Annex III, the provisions regarding technical documentation listed in Annex IV, the content of the EU declaration of conformity in Annex V, the provisions regarding the conformity assessment procedures in Annex VI and VII and the provisions establishing the high-risk AI systems to which the conformity assessment procedure based on assessment of the quality management system and assessment of the technical documentation should apply.
It is of particular importance that the Commission carry out appropriate consultations during its preparatory work, including at expert level, and that those consultations be conducted in accordance with the principles laid down in the Interinstitutional Agreement of 13 April 2016 on Better Law-Making58 . In particular, to ensure equal participation in the preparation of delegated acts, the European Parliament and the Council receive all documents at the same time as Member States’ experts, and their experts systematically have access to meetings of Commission expert groups dealing with the preparation of delegated acts.
(85) In order to ensure that the regulatory framework can be adapted where necessary, the power to adopt acts in accordance with Article 290 TFEU should be delegated to the Commission to amend the techniques and approaches referred to in Annex I to define AI systems, the Union harmonisation legislation listed in Annex II, the high-risk AI systems listed in Annex III, the provisions regarding technical documentation listed in Annex IV, the content of the EU declaration of conformity in Annex V, the provisions regarding the conformity assessment procedures in Annex VI and VII and the provisions establishing the high-risk AI systems to which the conformity assessment procedure based on assessment of the quality management system and assessment of the technical documentation should apply.
It is of particular importance that the Commission carry out appropriate consultations during its preparatory work, including at expert level, and that those consultations be conducted in accordance with the principles laid down in the Interinstitutional Agreement of 13 April 2016 on Better Law-Making58. Those consultations should involve the participation of a balanced selection of stakeholders, including consumer organisations, associations representing the persons concerned, representatives of businesses from different sectors and of different sizes, as well as researchers and scientists.
In particular, to ensure equal participation in the preparation of delegated acts, the European Parliament and the Council receive all documents at the same time as Member States’ experts, and their experts systematically have access to meetings of Commission expert groups dealing with the preparation of delegated acts.
Amendment 22 Proposal for a regulation Recital 86 a (new) Text proposed by the Commission Amendment
(86a) Given the rapid technological developments and the required technical expertise in conducting the assessment of high-risk AI systems, the delegation of powers and the implementing powers of the Commission should be exercised with as much flexibility as possible. The Commission should regularly review Annex III without undue delay, at least every six months, while consulting with the relevant stakeholders, including ethics experts, anthropologists, sociologists, mental health specialists and any other relevant scientists and researchers, as well as with parent associations.
Or. {EN}en Amendment 23 Proposal for a regulation Article 3 – paragraph 1 – point 34
Text proposed by the Commission Amendment
(34) ‘emotion recognition system’ means an AI system for the purpose of identifying or inferring emotions or intentions of natural persons on the basis of their biometric data;
(34) ‘emotion recognition system’ means an AI system for the purpose of identifying or inferring emotions, states of mind or intentions of natural persons on the basis of their biometric data or data relating to their physiological or behavioral characteristics; Or. {EN}en Amendment 24 Proposal for a regulation Article 3 – paragraph 1 – point 35
Text proposed by the Commission Amendment
(35) ‘biometric categorisation system’ means an AI system for the purpose of assigning natural persons to specific categories, such as sex, age, hair colour, eye colour, tattoos, ethnic origin or sexual or political orientation, on the basis of their biometric data;
(35) ‘biometric categorisation system’ means an AI system that uses biometric data, or other physical, physiological or behavioral data, for the purpose of assigning natural persons to specific categories, such as sex, age, hair colour, eye colour, tattoos, ethnic origin or sexual or political orientation, on the basis of their biometric data; Or. {EN}en Amendment 25 Proposal for a regulation Article 3 – paragraph 1 – point 36
Text proposed by the Commission Amendment
(36) ‘remote biometric identification system’ means an AI system for the purpose of identifying natural persons at a distance through the comparison of a person’s biometric data with the biometric data contained in a reference database, and without prior knowledge of the user of the AI system whether the person will be present and can be identified ;
(36) ‘remote biometric identification system’ means an AI system for the purpose of identifying natural persons at a distance through the comparison of a person’s biometric data with the biometric data contained in a reference database; Or. {EN}en Amendment 26 Proposal for a regulation Article 3 – paragraph 1 – point 39
Text proposed by the Commission Amendment
(39) ‘publicly accessible space’ means any physical place accessible to the public, regardless of whether certain conditions for access may apply;
(39) ‘publicly accessible space’ means any physical or virtual place accessible to the public, regardless of whether certain conditions for access may apply; Or. {EN}en Amendment 27 Proposal for a regulation Article 3 – paragraph 1 – point 44 a (new) Text proposed by the Commission Amendment
(44a) ‘education and training institutions’ means providers where people of different ages gain education and training, including preschools, childcare, primary schools, secondary schools, tertiary education providers, vocational education and training and any type of lifelong learning providers. Or. {EN}en Amendment 28 Proposal for a regulation Article 5 – paragraph 1 – point a
Text proposed by the Commission Amendment
(a) the placing on the market, putting into service or use of an AI system that deploys subliminal techniques beyond a person’s consciousness in order to materially distort a person’s behaviour in a manner that causes or is likely to cause that person or another person physical or psychological harm;
(a) the placing on the market, putting into service or use of an AI system that deploys techniques with the effect or likely effect of materially distorting a person’s behaviour in a manner that causes or is likely to cause that person or another person material or non-material harm including physical, psychological or economic harm; Or. {EN}en Amendment 29 Proposal for a regulation Article 5 – paragraph 1 – point b
Text proposed by the Commission Amendment
(b) the placing on the market, putting into service or use of an AI system that exploits any of the vulnerabilities of a specific group of persons due to their age, physical or mental disability, in order to materially distort the behaviour of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;
(b) the placing on the market, putting into service or use of an AI system that exploits any of the vulnerabilities of a person due to their known or predicted personality or social or economic situation or due to their age, physical or mental capacity, in order to materially distort the behaviour of a person in a manner that causes or is likely to cause that person or another person material or non-material harm, including physical, psychological or economic harm; Or. {EN}en Amendment 30 Proposal for a regulation Article 5 – paragraph 1 – point c – introductory part
Text proposed by the Commission Amendment
(c) the placing on the market, putting into service or use of AI systems by public authorities or on their behalf for the evaluation or classification of the trustworthiness of natural persons over a certain period of time based on their social behaviour or known or predicted personal or personality characteristics, with the social score leading to either or both of the following:
(c) the placing on the market, putting into service or use of AI systems for the evaluation or classification of the trustworthiness of natural persons based on their social behaviour or known or predicted personal or personality characteristics; Or. {EN}en Amendment 31 Proposal for a regulation Article 5 – paragraph 1 – point c – point i
Text proposed by the Commission Amendment
(i) detrimental or unfavourable treatment of certain natural persons or whole groups thereof in social contexts which are unrelated to the contexts in which the data was originally generated or collected; deleted Or. {EN}en Amendment 32 Proposal for a regulation Article 5 – paragraph 1 – point c – point ii
Text proposed by the Commission Amendment
(ii) detrimental or unfavourable treatment of certain natural persons or whole groups thereof that is unjustified or disproportionate to their social behaviour or its gravity; deleted Or. {EN}en Amendment 33 Proposal for a regulation Article 5 – paragraph 1 – point d – introductory part
Text proposed by the Commission Amendment
(d) the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement, unless and in as far as such use is strictly necessary for one of the following objectives:
(d) the use of ‘real-time’ and ‘post’ remote biometric identification systems of natural persons in publicly accessible spaces. Or. {EN}en Amendment 34 Proposal for a regulation Article 5 – paragraph 1 – point d – point i
Text proposed by the Commission Amendment
(i) the targeted search for specific potential victims of crime, including missing children; deleted Or. {EN}en Amendment 35 Proposal for a regulation Article 5 – paragraph 1 – point d – point ii Text proposed by the Commission Amendment
(ii) the prevention of a specific, substantial and imminent threat to the life or physical safety of natural persons or of a terrorist attack; deleted Or. {EN}en Amendment 36 Proposal for a regulation Article 5 – paragraph 1 – point d – point iii
Text proposed by the Commission Amendment
(iii) the detection, localisation, identification or prosecution of a perpetrator or suspect of a criminal offence referred to in Article 2(2) of Council Framework Decision 2002/584/JHA62 and punishable in the Member State concerned by a custodial sentence or a detention order for a maximum period of at least three years, as determined by the law of that Member State. deleted 62 Council Framework Decision 2002/584/JHA of 13 June 2002 on the European arrest warrant and the surrender procedures between Member States (OJ L 190, 18.7.2002, p. 1).
Or. {EN}en Amendment 37 Proposal for a regulation Article 5 – paragraph 2
2The use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement for any of the objectives referred to in paragraph 1 point d) shall take into account the following elements:
deleted
(a) the nature of the situation giving rise to the possible use, in particular the seriousness, probability and scale of the harm caused in the absence of the use of the system;
(b) the consequences of the use of the system for the rights and freedoms of all persons concerned, in particular the seriousness, probability and scale of those consequences.
In addition, the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement for any of the objectives referred to in paragraph 1 point d) shall comply with necessary and proportionate safeguards and conditions in relation to the use, in particular as regards the temporal, geographic and personal limitations.
Or. {EN}en Amendment 38 Proposal for a regulation Article 5 – paragraph 3 Text proposed by the Commission Amendment
3As regards paragraphs 1, point (d) and 2, each individual use for the purpose of law enforcement of a ‘real-time’ remote biometric identification system in publicly accessible spaces shall be subject to a prior authorisation granted by a judicial authority or by an independent administrative authority of the Member State in which the use is to take place, issued upon a reasoned request and in accordance with the detailed rules of national law referred to in paragraph 4. However, in a duly justified situation of urgency, the use of the system may be commenced without an authorisation and the authorisation may be requested only during or after the use.
deleted The competent judicial or administrative authority shall only grant the authorisation where it is satisfied, based on objective evidence or clear indications presented to it, that the use of the ‘real-time’ remote biometric identification system at issue is necessary for and proportionate to achieving one of the objectives specified in paragraph 1, point (d), as identified in the request. In deciding on the request, the competent judicial or administrative authority shall take into account the elements referred to in paragraph 2.
Or. {EN}en Amendment 39 Proposal for a regulation Article 5 – paragraph 4 Text proposed by the Commission Amendment
4A Member State may decide to provide for the possibility to fully or partially authorise the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement within the limits and under the conditions listed in paragraphs 1, point (d), 2 and 3. That Member State shall lay down in its national law the necessary detailed rules for the request, issuance and exercise of, as well as supervision relating to, the authorisations referred to in paragraph 3. Those rules shall also specify in respect of which of the objectives listed in paragraph 1, point (d), including which of the criminal offences referred to in point (iii) thereof, the competent authorities may be authorised to use those systems for the purpose of law enforcement.
deleted Or. {EN}en Amendment 40 Proposal for a regulation Article 7 – paragraph 1 – introductory part
1The Commission is empowered to adopt delegated acts in accordance with Article 73 to update the list in Annex III by adding high-risk AI systems where both of the following conditions are fulfilled:
1The Commission is empowered to adopt delegated acts in accordance with Article 73 to update the list in Annex III by adding high-risk AI systems where the AI systems pose a risk of harm to the health and safety, or a risk of adverse impact on fundamental rights, that is, in respect of its severity and probability of occurrence.
Or. {EN}en Amendment 41 Proposal for a regulation Article 7 – paragraph 1 – point a
Text proposed by the Commission Amendment
(a) the AI systems are intended to be used in any of the areas listed in points 1 to 8 of Annex III; deleted Or. {EN}en Amendment 42 Proposal for a regulation Article 7 – paragraph 2 – introductory part
2When assessing for the purposes of paragraph 1 whether an AI system poses a risk of harm to the health and safety or a risk of adverse impact on fundamental rights that is equivalent to or greater than the risk of harm posed by the high-risk AI systems already referred to in Annex III, the Commission shall take into account the following criteria:
2When assessing for the purposes of paragraph 1 whether an AI system poses a risk of harm to the health and safety or a risk of adverse impact on fundamental rights, society or the environment that is equivalent to or greater than the risk of harm posed by the high-risk AI systems already referred to in Annex III, the Commission shall take into account the following criteria:
Or. {EN}en Amendment 43 Proposal for a regulation Article 7 – paragraph 2 a (new) Text proposed by the Commission Amendment
2a. The Commission shall conduct the assessment referred to in paragraph 2 without undue delay and at least every six months, under the consultation conditions laid down in this Regulation, in particular in Article 73. Or. {EN}en Amendment 44 Proposal for a regulation Article 73 – paragraph 3 a (new) Text proposed by the Commission Amendment
3a. Before adopting a delegated act, the Commission shall consult with the relevant institutions and stakeholders in accordance with the principles laid down in the Interinstitutional Agreement of 13 April 2016 on Better Law-Making. Or. {EN}en Amendment 45 Proposal for a regulation Annex III – paragraph 1 – point 1 – point a a (new) Text proposed by the Commission Amendment
(aa) AI systems which use data relating to physical, physiological or behavioural characteristics, including biometric data; Or. {EN}en Amendment 46 Proposal for a regulation Annex III – paragraph 1 – point 1 – point a b (new) Text proposed by the Commission Amendment
(ab) AI systems that are intended be used for the non-remote biometric identification of natural persons in publicly accessible spaces, as well as in workplaces and in education and training institutions. Or. {EN}en Amendment 47 Proposal for a regulation Annex III – paragraph 1 – point 3 – introductory part
3Education and vocational training:
3Education and training:
Or. {EN}en Amendment 48 Proposal for a regulation Annex III – paragraph 1 – point 3 – point a
Text proposed by the Commission Amendment
(a) AI systems intended to be used for the purpose of determining access or assigning natural persons to educational and vocational training institutions;
(a) AI systems intended to be used for the purpose of determining access or assigning natural persons to educational and training institutions; Or. {EN}en Amendment 49 Proposal for a regulation Annex III – paragraph 1 – point 3 – point b
Text proposed by the Commission Amendment
(b) AI systems intended to be used for the purpose of assessing students in educational and vocational training institutions and for assessing participants in tests commonly required for admission to educational institutions.
(b) AI systems intended to be used for the purpose of assessing students in educational and training institutions and for assessing participants in tests commonly required for admission to educational institutions; Or. {EN}en Amendment 50 Proposal for a regulation Annex III – paragraph 1 – point 3 – point b a (new) Text proposed by the Commission Amendment
(ba) AI systems intended to be used for monitoring and detecting prohibited behaviour of students during tests at education and training institutions; Or. {EN}en Amendment 51 Proposal for a regulation Annex III – paragraph 1 – point 3 – point b b (new) Text proposed by the Commission Amendment
(bb) AI systems intended to be used for the purpose of determining the study program or areas of study to be followed by students in education and training institutions; Or. {EN}en Amendment 52 Proposal for a regulation Annex III – paragraph 1 – point 3 a (new) Text proposed by the Commission Amendment
3a. Media and culture
(a) AI systems used by news media outlets intended to be used to create or disseminate machine-generated news articles;
(b) AI systems used by audiovisual media services intended to make recommendations to natural persons or to prioritize or rank specific audiovisual content in the online interface of the audiovisual media service. Or. {EN}en Amendment 53 Proposal for a regulation Annex III – paragraph 1 – point 4 – point b
Text proposed by the Commission Amendment
(b) AI intended to be used for making decisions on promotion and termination of work-related contractual relationships, for task allocation and for monitoring and evaluating performance and behavior of persons in such relationships.
(b) AI intended to be used for making decisions on promotion and termination of work-related relationships, for task allocation, for monitoring compliance with workplace rules and for monitoring and evaluating performance and behavior of persons in such relationships. Or. {EN}en </RepeatBlock-Amend>
The following list is drawn up on a purely voluntary basis under the exclusive responsibility of the rapporteur. The rapporteur has received input from the following entities or persons in the preparation of the draft opinion: Entity and/or person AccessNow Baptiste Caramiaux, CNRS researcher, Sorbonne Université in Paris
Defenddigitalme EDRi European Disability Forum Huawei International Federation of Library Associations and Institutions Joanna Bryson, Professor of Ethics and Technology, Hertie School Liber Europe Maldita.es Martin Senftlebe, Professor of Intellectual Property Law, University of Amsterdam Meaning Processing Ltd.
Saidot Ltd. Scio.cz