

# Event Details

- **Event Name**: Zero-shot Benchmarking: Flexible and Scalable Automatic Evaluation of LLMs
- **Event Start and End Date**: Tue, 07 Apr, 2026 at 01:00 pm – Tue, 07 Apr, 2026 at 02:00 pm (+01:00)
- **Event Description**: Priberam Machine Learning Lunch SeminarAbout this EventAbstract:As language models improve and become capable of performing more complex tasks across modalities, evaluating them automatically becomes increasingly challenging. Developing strong and robust task-specific automatic metrics gets harder, and human-annotated test sets -- which are expensive to create -- saturate more quickly. A compelling alternative is to design reliable strategies to automate the creation of test data and evaluation, but previous attempts either rely on pre-existing data, or focus solely on individual tasks. We present Zero-shot Benchmarking (ZSB), a framework for creating high-quality benchmarks for any task by leveraging language models for both synthetic test data creation and evaluation. ZSB is simple and flexible: it requires only the creation of a prompt for data generation and one for evaluation; it is scalable to tasks and languages where collecting real-world data is costly or impractical; it is model-agnostic, allowing the creation of increasingly challenging benchmarks as models improve. To assess the effectiveness of our framework, we create benchmarks for five text-only tasks and a multi-modal one: general capabilities in four languages (English, Chinese, French, and Korean), translation, and general vision-language capabilities in English. We then rank a broad range of open and closed systems on our benchmarks. ZSB rankings consistently correlate strongly with human rankings, outperforming widely-adopted standard benchmarks. Through ablations, we find that strong benchmarks can be created with open models, and that judge model size and dataset variety are crucial drivers of performance. We release all our benchmarks, and code to reproduce our experiments and to produce new benchmarks.Bio:José Pombal is a Senior Research Scientist at Sword Health and a PhD student at IST, with his research focusing on the automatic evaluation of LLMs and their application to mental health therapy.
- **Event URL**: https://allevents.in/lisboa/zero-shot-benchmarking-flexible-and-scalable-automatic-evaluation-of-llms/100001985928064300
- **Event Categories**: health-wellness, Entertainment, Workshops, performances
- **Interested Audience**: 
  - total_interested_count: 0

## Ticket Details

- **Ticket URL**: https://consumer.pxf.io/c/1339408/3151484/39691?partnerpropertyid=1191892&MediaPartnerPropertyId=1191892&svlink=14105511&level=1&u=https%3A%2F%2Fwww.eventbrite.com%2Fe%2Fzero-shot-benchmarking-flexible-and-scalable-automatic-evaluation-of-llms-tickets-1985928064300
- **Ticket Price Range**: min: 0, max: 0, currency: EUR

## Event venue details

- **city**: Lisboa
- **state**: LI
- **country**: Portugal
- **location**: Instituto Superior Técnico, Anfiteatro PA2
- **lat**: 38.7368192
- **long**: -9.138705
- **full address**: Instituto Superior Técnico, Anfiteatro PA2, 1 Avenida Rovisco Pais, Lisboa, Portugal

## Event Organizer details

- **affiliate_id**: 10
- **organizer**: Priberam Labs (https://allevents.in/org/priberam-labs/23609284)

## Event gallery

- **Alt text**: banner
  - **Image URL**: https://cdn-az.allevents.in/events8/banners/2aa2d1814085083cd7f17eb7cdfac177489c39bdf50322a6918061b2806a6be3-rimg-w1173-h704-dcfefeff-gmir.jpg?v=1774542865
- **Alt text**: thumbnail
  - **Image URL**: https://cdn-az.allevents.in/events8/banners/befb93a7cd6f560040effb8b9086dbbea38f50b2e42e6d3ecbb35e794473cb73-rimg-w400-h400-dcfefffe-gmir.jpg?v=1774542865

## FAQs

- **Q**: When is the event happening?
  - **A:** Tue, 07 Apr, 2026 at 01:00 pm (+01:00)
- **Q**: Where is the event happening?
  - **A:** Instituto Superior Técnico, Anfiteatro PA2, 1 Avenida Rovisco Pais, Lisboa, Portugal
- **Q**: What is the ticket price?
  - **A:** starting from 0 up to 0 EUR
- **Q**: Who is organizing the event?
  - **A:** Priberam Labs
- **Q**: What type of event is this?
  - **A:** health-wellness, Entertainment, Workshops, performances
- **Q**: Where can I find ticket details about Zero-shot Benchmarking: Flexible and Scalable Automatic Evaluation of LLMs ?
  - **A:** You can find ticket details about this event on AllEvents.

## Structured Data (JSON-LD)

```json
[
    {
        "@context": "https://schema.org",
        "@type": "Event",
        "name": "Zero-shot Benchmarking: Flexible and Scalable Automatic Evaluation of LLMs",
        "image": "https://cdn-az.allevents.in/events8/banners/2aa2d1814085083cd7f17eb7cdfac177489c39bdf50322a6918061b2806a6be3-rimg-w1173-h704-dcfefeff-gmir.jpg?v=1774542865",
        "startDate": "2026-04-07T13:00:00+01:00",
        "endDate": "2026-04-07T14:00:00+01:00",
        "url": "https://allevents.in/lisboa/zero-shot-benchmarking-flexible-and-scalable-automatic-evaluation-of-llms/100001985928064300",
        "eventStatus": "https://schema.org/EventScheduled",
        "location": {
            "@type": "Place",
            "name": "Instituto Superior Técnico, Anfiteatro PA2",
            "address": {
                "@type": "PostalAddress",
                "streetAddress": "1 Avenida Rovisco Pais",
                "postalCode": "1049-001",
                "addressLocality": "Lisboa",
                "addressRegion": "LI",
                "addressCountry": "PT"
            },
            "geo": {
                "@type": "GeoCoordinates",
                "latitude": "38.7368192",
                "longitude": "-9.138705"
            }
        },
        "eventAttendanceMode": "https://schema.org/OfflineEventAttendanceMode",
        "description": "Priberam Machine Learning Lunch Seminar",
        "offers": [
            {
                "@type": "AggregateOffer",
                "availability": "https://schema.org/InStock",
                "priceCurrency": "EUR",
                "availabilityStarts": "2026-03-26",
                "availabilityEnds": "2026-04-07T14:00:00+01:00",
                "validFrom": "2026-03-26",
                "lowPrice": "0.00",
                "highPrice": "0.00",
                "price": "0.00",
                "url": "https://allevents.in/lisboa/zero-shot-benchmarking-flexible-and-scalable-automatic-evaluation-of-llms/100001985928064300"
            },
            {
                "@type": "Offer",
                "availability": "https://schema.org/InStock",
                "priceCurrency": "EUR",
                "availabilityStarts": "2026-03-26",
                "availabilityEnds": "",
                "validFrom": "2026-03-26",
                "url": "https://allevents.in/lisboa/zero-shot-benchmarking-flexible-and-scalable-automatic-evaluation-of-llms/100001985928064300",
                "price": 0,
                "name": "Normal"
            },
            {
                "@type": "Offer",
                "availability": "https://schema.org/InStock",
                "priceCurrency": "EUR",
                "availabilityStarts": "2026-03-26",
                "availabilityEnds": "",
                "validFrom": "2026-03-26",
                "url": "https://allevents.in/lisboa/zero-shot-benchmarking-flexible-and-scalable-automatic-evaluation-of-llms/100001985928064300",
                "price": 0,
                "name": "Veggy"
            },
            {
                "@type": "Offer",
                "availability": "https://schema.org/InStock",
                "priceCurrency": "EUR",
                "availabilityStarts": "2026-03-26",
                "availabilityEnds": "",
                "validFrom": "2026-03-26",
                "url": "https://allevents.in/lisboa/zero-shot-benchmarking-flexible-and-scalable-automatic-evaluation-of-llms/100001985928064300",
                "price": 0,
                "name": "No meal"
            }
        ],
        "organizer": [
            {
                "@type": "Organization",
                "name": "Priberam Labs",
                "url": "https://allevents.in/org/priberam-labs/23609284"
            }
        ]
    }
]
```