As artificial intelligence (AI) becomes increasingly integrated into various aspects of daily life, the need for AI literacy continues to grow. Accordingly, both educators and students must develop a critical understanding of AI systems and acquire the skills to apply them responsibly. However, AI literacy education is associated with barriers for teachers. This study examines teacher perspectives on AI literacy education through focus group discussions and single interviews with 18 primary school teachers from all of Switzerland (French- Italian- and German-speaking regions). For the analysis of the interviews, a summarizing qualitative content analysis was applied, incorporating barriers to technology integration and AI literacy education taken from the research literature. The analysis sheds light on barriers to AI literacy education in the context of early primary education. The results indicate that barriers identified in the literature were also present in the context of primary school teachers in Switzerland. Moreover, it became apparent that misconceptions about the scope of AI are closely linked to a lack of ideas for pedagogical implementation of AI literacy. Additionally, teachers’ concerns about AI’s high transformative potential might be linked with teachers’ hesitation to integrate AI literacy education in their classroom. Implications for teacher professional development (PD) are derived.