Must-Have Prompt Engineering Skills for 2024
The role of prompt engineer has attracted massive interest ever since Business Insider released an article last spring titled “AI ‘Prompt Engineer Jobs: $375k Salary, No Tech Backgrund Required.”
This article lit up the internet and sparked widespread discussion by highlighting the potential for high salaries even for those without traditional tech backgrounds. While many of us dream of having a job in AI that doesn’t require knowing AI tools and skillsets, that’s not actually the case. It turns out that the role of a Prompt Engineer is not simply typing questions into a prompt window.
Thus while crafting clever prompts for chatbots might be part of the picture, the prompt engineer role is far more intricate. Think of them as architects of language-driven AI. They design intricate sequences of prompts, leveraging their knowledge of AI, machine learning, and data science to guide powerful LLMs (Large Language Models) towards complex tasks.
To better understand the role, we took a sampling of about ~430 Prompt Engineering Job descriptions to see what employers were looking for. We don’t claim this is a definitive analysis but rather a rough guide due to several factors:
- Job descriptions show lagging indicators of in-demand prompt engineering skills, especially when viewed over the course of 9 months.
- Employers and hiring managers tend to favor more established skills vs. the latest tools and platforms.
- The definition of a particular job role is constantly in flux and varies from employer to employer. This is especially true for new and evolving roles such as Prompt Engineering.
- This is not a definitive list and is most likely biased by its collection source. Many tools and topics we know are popular have yet to make it here. Treat this as a guide and know it is not definitive.
Since the job is new and many candidates aren’t totally sure what the job looks like just yet, we looked at existing prompt engineering job descriptions and organized the must-have prompt engineering skills, platforms, and use cases that you need to have to stand out.
We also examined the results to gain a deeper understanding of why these prompt engineering skills and platforms are in demand for the role of Prompt Engineer, not to mention machine learning and data science roles.
Prompt Engineering Skills
Core Knowledge of Artificial Intelligence, Machine Learning, Data Science, and Neural Networks
Foundational Skills
Unsurprisingly, employers are seeking candidates with strong foundational skills in artificial intelligence, machine learning, deep learning, and neural networks, as these are essential for understanding and working with the algorithms that power prompt engineering. Knowledge in these areas enables prompt engineers to understand the mechanics of language models and how to apply them effectively.
Data Science
Knowing the ins and outs of data science encompasses the ability to handle, analyze, and interpret data, which is required for training models and understanding their outputs. Prompt engineering often involves a lot of experimentation to find the best way to phrase prompts. Data science methodologies and skills can be leveraged to design these experiments, analyze results, and iteratively improve prompt strategies. Using skills such as statistical analysis and data visualization techniques, prompt engineers can assess the effectiveness of different prompts and understand patterns in the responses.
EVENT — ODSC East 2024
In-Person and Virtual Conference
April 23rd to 25th, 2024
Join us for a deep dive into the latest data science and AI trends, tools, and techniques, from LLMs to data analytics and from machine learning to responsible AI.
Specialized Prompt Engineering Skills, Fine-Tuning, Inference, Embeddings, Scaling Laws, and GenAI
Prompt Engineering
No surprise here as it is the skill specifically named after the role.
Fine-Tuning
Perhaps one of the most important skills for the role is considered essential. Fine-tuning involves modifying pre-trained models for specific tasks that are custom to a particular project’s needs, such as summarizing clinical trial data. Fine-tuning is important for applying domain-specific knowledge to an existing LLM which provides better performance and prompt results
Inference Efficiency
An emergent skill in late 2023, its inclusion speaks to its importance. This skill focuses on minimizing the resources and time required for an LLM to generate output based on your prompts. Given the high cost of fine-tuning LLMs, expect a lot of focus on this area in 2024
Embeddings
Embeddings allow you to represent how linguistic data is represented within models, a key concept in understanding and manipulating model inputs and outputs. LLMs like GPT-4 and LLaMA 2 generate word embeddings dynamically based on the context of each word within a sentence or paragraph. Some LLMs also offer methods to produce embeddings for entire sentences or documents, capturing their overall meaning and semantic relationships. These outputs, stored in vector databases like Weaviate, allow Prompt Enginers to directly access these embeddings for tasks like semantic search, similarity analysis, or clustering.
Fine-tuning a pre-trained LLM on a dataset tailored to a service or task will generate embeddings that better reflect domain-specific knowledge or nuances services like question answering or sentiment analysis. Retrieval Augmented Generation (RAG) system can also use Vector databases to act as long-term memory for LLMs via embeddings, storing and retrieving relevant information based on semantic similarity. This enhances the context awareness and factual accuracy of LLM outputs.
Generative AI
Generative AI is another crucial skill for the role of prompt engineering, as it encompasses the core ability to leverage AI to create new content, whether it be text, images, or other forms of media. GANs, introduced in 2014 paved the way for GenAI with models like Pix2pix and DiscoGAN. It wasn’t until the release of more advanced generative models such as variational autoencoders, transformer-based models, and autoregressive models that lead generative AI exemplified by the release of GPT-3, DALL-E, and Midjourney. Text-to-image and text-to-video models are rapidly improving in accuracy and resolution so prompt engineering will be in demand to tune models that can create an expanding range of services from product design, advertising, user interfaces, and even advanced visualization.
Scaling Laws
These are more advanced and emerging areas in AI. Scaling laws refer to how the performance of AI models scales with size. There are various scaling laws including the influential Chinchila scaling laws as outlined in this DeepMind paper. These laws will have an outsized impact on how far LLMs can progress in the new feature and something prompt engineers will be monitoring closely.
Leaving aside the more established skills here’s a visual look at the newer skills
Natural Language Processing (NLP), Tokenization, Transformers, Representation Learning and Knowledge Graphs
NLP (Natural Language Processing)
The NLP engineer can be considered a precursor to the Promt Engineer. NLP skills have long been essential for dealing with textual data. These skills help with designing and optimizing prompts, understanding model outputs, and tailoring applications to specific language tasks.
Tokenization & Transformers
These are specific techniques in NLP and popularized by LLMs. Tokenization involves converting text into a format understandable by models. Tokens are vital to how LLMs understand and process information. They act as the building blocks for everything an LLM does, like comprehending text, generating responses, and completing tasks. During training, LLMs adjust their internal parameters based on how well they predict the next token in a sequence. Thus understanding the efficient use of tokens and resource management is necessary to process, fine-tune, or train large amounts of text and perform complex tasks on even limited hardware resources.
Transformers
Transformers is a well-known model architecture used in most modern LLMs. Pre-trained LLMs contain general structure, knowledge, understanding of, and patterns of language. It’s transfer learning that allows prompt engineers to fine-tune these models for specific domains and tasks, increasing their accuracy and relevance for those contexts. Additionally, prompt engineers may not have access to large amounts of domain-specific data. Transfer learning allows prompt engineers to make the most of limited data by using pre-trained models as a starting point. This empowers them to work with smaller datasets and still achieve good results for specific tasks.
Representation Learning & Knowledge Graphs
This involves understanding how to represent linguistic data in a way that’s useful for AI models and leveraging structured knowledge sources. For a prompt engineer, representation learning reveals how the LLM encodes information from text and data into its internal structures. Knowledge graphs provide interconnected networks of relationships. Knowledge graphs can supplement and model’s understanding of the world and improve its ability to generate consistent and accurate responses.
Data, Engineering, and Programming Skills
Programming
Despite the rise of no-code platforms and AI code assistance, programming skills are still essential for training and fine-tuning LLM models, scripting for data processing, and integrating models into applications.
Data Engineering
A job role in its own right, this involves managing the modern data stack and structuring data and workflow pipelines — crucial for preparing data for use in training and running AI models.
Data Analysis
Data analysis is often overlooked, but it’s still an essential skill for interpreting results from AI models and for the iterative process of improving prompt responses.
Prompt Engineering Platforms
LLM Platforms: ChatGpt, GPT-4, LLama 2, Stable Diffusion, and BERT
ChatGPT
OpenAI’s ChatGPT was one of the most popular apps in history, so it’s no surprise that the suite of API models including GPT-3.5 series (Davinci, etc), GPT-4, and GPT-4 Turbo are immensely popular. As the most popular platform, employers view it as pivotal in prompt engineering. It’s an advanced language model that allows prompt engineers to create complex, context-aware prompts capable of generating sophisticated responses. The latest API model (at the time of writing) is the GPT-4 Tubo API which offers multi-step instructions, a larger context window of up to 128k tokens (roughly 300 pages of text), and sophisticated multimodal abilities for image processing such as image captioning, visual content analysis,
LLama-2
Despite only being released mid-year (July 18, 2023), it quickly became a go-to platform for prompt engineers. Unlike closed models such as GPT-4 and Anthropic’s Claude, the LLaMA series of models are open source and available to researchers and developers. The open nature of LLaMA allows prompt engineers to fine-tune it for specific tasks, tailoring its response to their prompts and desired outputs. This level of control unlocks creative possibilities and with access to the model’s code and training data, prompt engineers can delve deeper into its inner workings and develop innovative new applications.
Stable Diffusion
Text-to-image models like DALL-E and Stable Diffusion were a breakout success in 2023 and captured employers’ attention thanks to their ability to generate realistic images from limited text descriptions. Stable Diffusion seems favored, perhaps due to it being largely an open-source model. From a platform perspective, this has led to a rich ecosystem surrounding the model, including codebases, libraries, user interfaces, and online communities. While technically an LLM due to its limited language understanding, these text-to-image/video models can be seen as carving their own niche between LLMs and LVMs (Large Vision Models).
BERT
While technically not an LLM (pre-dates LLMs), due to its 360 million parameters vs the (supposed) 1.76 trillion parameters in GPT-4 it is built using the transformer encoder architecture that continues to be used for many LLMs that followed it. It’s a pre-trained model capable of various tasks like text classification, question answering, and sentiment analysis. This versatility allows prompt engineers to adapt it to different projects and needs.
Open Source ML/DL Platforms: Pytorch, Tensorflow, and scikit-learn
Hiring managers continue to favor the most popular open-source machine/deep learning platforms including Pytorch, Tensorflow, and scikit-learn. Each continues to evolve and remain quite popular across multiple AI job roles including new roles such as prompt engineering. While not designed specifically for working with pre-trained models and prompt engineering, these platforms’ rich feature list flexibility and power make for enduring popularity. New features have been added to support pre-trained models and integration with libraries like Hugging Face’s Transformers.
For the role of a prompt engineer, platforms like PyTorch, TensorFlow, and scikit-learn can also be used for various downstream tasks in AI and machine learning. They typically involve refining and applying AI models to specific applications and tasks including model evaluation and selection.
LangChain and Agents
LangChain is popular with prompt engineers due to its being one of the first open-source libraries for building smart LLM agents by chaining custom prompts and allowing complex task execution with custom and personalized responses. Its many unique features include:
- Chain Prompts and Agents: A key feature of any agent platform is the ability to create whole sequences of prompts, thus allowing agents to handle complex tasks involving multiple steps and decisions. LangChain’s architecture supports the integration of chains and agents, allowing complex task executions.
- Prompt Templates: These reusable templates provide a structured way to create and manage prompts for language models through its PromptTemplate object. These templates simplify the process of formulating queries, etc.
- Memory Feature: Capturing past chats is essential, so LangChain includes a “Memory” feature, which is crucial for applications that involve chat since earlier conversations need to be recalled for follow-up questions. This allows a contextual understanding of previous messages and provides stateful behavior
With frequent feature releases and being an open-source project, LangChain is poised to be a favored platform for prompt engineers. However, users do note a steep learning curve and performance and scalability issues. Thus open source alternatives like LlamaIndex are emerging.
Prompt Engineer’s Established Toolkit: Hugging Face, NLTK, Spacy, Sagemaker, Spark, Databricks, Kubernetes, and Shap
While not the newest of platforms, many of these tools are essential to the role of prompt engineering
- Hugging Face: Offers an ever-expanding ecosystem of pre-trained LLMs, libraries, and tools for fine-tuning, crafting prompts, and analyzing results
- NLTK: Long an NLP engineer’s favorite, NLTK provides foundational language processing tools for pre-processing text, extracting features, and analyzing linguistic structures.
- Spacy: Another firm favorite, Spacy can be used for tokenization, part-of-speech tagging, and named entity recognition, aiding in structuring prompts accurately.
- Apache Spark: Enables distributed data processing and is essential for handling massive datasets and scaling projects efficiently.
- Sagemaker: Provides a cloud-based platform for fine-tuning and deploying LLM models, simplifying workflow and resource management.
- Databricks: Powered by Apache Spark, Databricks is a unified data processing and analytics platform, facilitates data preparation, can be used for integration with LLMs, and performance optimization for complex prompt engineering tasks.
- Kubernetes: A long-established tool for containerized apps. For prompt engineers, it can be used for the deployment and orchestration of LLM applications.
- Shap: Currently LLMs are not directly explainable in the same way as simpler machine learning models due to their complexity, size, and the black box nature of closed source models. However, tools like Shap’s text Explainer still can analyze text and model behavior, revealing what drives model responses and identifying potential biases.
Prompt Engineering: Favored Programming Languages
While Python’s dominance comes as no surprise, the presence of R and other languages raises an eyebrow. Perhaps it’s down to some hiring managers’ preferences or established code bases for integration.
Python
Python’s prominence is expected. Most popular LLMs, such as OpenAI’s API and tools like LangChain and Hugging Face transformers, offer readily available Python libraries and tutorials. Additionally, prompt engineering relies heavily on machine learning tasks like fine-tuning, bias detection, and performance evaluation. Python boasts a vast ecosystem of libraries like TensorFlow, PyTorch, Pandas, NumPy, and Scikit-learn, empowering prompt engineers to handle data wrangling and analysis seamlessly.
R
The presence of R is somewhat surprising given its traditional association with statistical analysis rather than AI, but there are reasons for its inclusion including its use. These include a wide user base and a growing set of machine learning libraries and interfaces. R also excels in data analysis and visualization, which are important in understanding the output of LLMs and in fine-tuning prompt strategies.
SQL
SQL’s importance stems from its role in data handling, and prompt engineers often need to query and manage large datasets, for which SQL is the standard language.
Prompt Engineering Cloud Services
While some prompt engineering jobs may just involve local use, it’s more likely that applications will be built on the cloud, requiring knowledge of developing and programming with cloud-based tools. While AWS is usually the winner when it comes to data science and machine learning, it’s Microsoft Azure that’s taking the lead for prompt engineering job descriptions.
Azure jumps out as the prompt engineer’s go-to cloud service as Azure has direct access to OpenAI’s LLMs like GPT and Codex, and this will allow prompt engineers to interact directly with these models through APIs and SDKs without having to dive into the underlying structure.
You may be expected to use other cloud platforms like AWS, GCP, and others, so don’t neglect them and at least be vaguely familiar with how they work.
How to Learn These Prompt Engineering Skills
We just listed off quite a lot of prompt engineering skills , platforms, and use cases that many prompt engineering job descriptions are calling for, so it may be a bit difficult to know where to start. At ODSC East this April 23rd to 25th, we’ll have two tracks where you can learn more about prompt engineering — one for NLP & LLMs, and one for Generative AI.
While we’re still in the early stages of planning, you can subscribe to our newsletter to be the first to hear about all sessions related to prompt engineering, LLMs, and generative AI.
Originally posted on OpenDataScience.com
Read more data science articles on OpenDataScience.com, including tutorials and guides from beginner to advanced levels! Subscribe to our weekly newsletter here and receive the latest news every Thursday. You can also get data science training on-demand wherever you are with our Ai+ Training platform. Interested in attending an ODSC event? Learn more about our upcoming events here.
