Deep Dive: Mike Blumenthal on the New Image Results Landscape in Local Search
Mike Blumenthal

Images everywhere!  Join Mike Blumenthal and Carrie Hill as they discuss the new frontier of images that are showing up in local search results. From 3-pack query-matched images, thumbnail images, site link images and “search by image” to taking a look at Google’s Vision AI tool – we’re going to tell you what the opportunities lie – and give a few tactics for being early to market with those opportunities for your clients or business!

Here are the links we reference:

Carrie: Hi everybody. Welcome. We’re going to record a deep dive video today. I’m so happy to have Mike with me. We’ve been seeing a lot of volatility or change in how images are presented and used in local search and the search engine results. And I thought it would be great to have Mike explain what he’s seen in his research. We’re going to talk about what the landscape looks like now, the history behind how we got there. Stick till the end because we’re going to have some tactics for how you can take advantage. So, I guess where we start, Mike, is like, what’s going on now? Do you want to show your screen or you want to talk for a few minutes?

Mike: Sure. So, imagery is critical to search. It’s critical to users converting. We’ve always known that Google has never wanted to show the images that you wanted to show in the local pack. And now we’re actually starting to gain insight into some of their thinking behind that. And I think that, by studying this, I think we can improve our websites, improve our Google My Business and improve conversions as well. And I think there’s both interesting technical things to look at as well some interesting tactical ones. And what got us going on this whole thread was some articles I wrote about the new search by photos and then some… What’s Andy’s last name?

Carrie: Andy Simpson.

Mike: Andy Simpson shared with us the vision AI features in Google where they expose their understanding of images to the API. And with all of that, we’re starting to develop a completer picture of the role that images are playing. Also, at LocalU Advanced, David Mihm showed some interesting research that showed that image viewing was going way up. Google has long encouraged businesses to post more images. And if you put two and two together, it appears that Google’s understanding of images has gotten so sophisticated that it’s now influencing their understanding of what your business does, what it sells, and thus that increase in image views coming from you uploading more images is likely also increasing the reach of your GMB listing. So, there’s a lot of stuff to parse through and think about here. And so, we’re going to switch off our video and go to some screen shares, right?

Carrie: Yeah, great. I think that, Google’s been giving us clues. That’s their MO, right? So, they say, “Hey, use this. Hey, use this. Hey, use this.” And then all of a sudden it becomes a big deal. And we’ve been getting emails from Google My Business saying, “Hey, your photos got this much engagement. Upload more photos for, what, two years probably?”

Mike: A year longer, yeah.

Carrie: Or longer. So, I think that, they’ve been encouraging people to upload those photos but now they’re rewarding you for being very image-centric, not only in your Google My Business but on your website as well. So I think that this is a building trend that is…I don’t want to call it coming to ahead because I think it can even grow more, but I think it’s bubbling to the top as far as how much… I mean, you’re going to show in the screen share how much real estate you can own with great image search.

Mike: Right, exactly. So, it got brought ahead last week when we noticed this new search paradigm which was search by photos, which is a mini-Carousel that it resides below the ads, below the pack, and potentially below the organic. Although it can, depending on its bit, it might…depending on the other things that are in the pack, it may show higher, it may show right up under the pack. But Search My Photos is this new search paradigm where what we discerned was that Google will change these photos based on the search query.

So, the search query was “engagement rings Williamsville.” And up come images of engagement rings that show. Here’s a search query for jewelry. You can see Barbara Oliver, what you saw in the other one. We’re now seeing a pendant instead of an engagement ring. And here’s a search for earrings. And, again, you see Barbara Oliver now showing earrings. So, what we’re seeing here is that the images are reflecting the search query in a very visual, almost Instagrammy way. And, obviously, this is somewhat redundant from the context of the pack. But there’s several things about it.

One is that it’s obviously, as Carrie points out, taking up a fair bit of real estate. But, two, it’s a very immerse…what Google calls immersive search. By that, I mean that when you click on it, it doesn’t take you off to a website. It takes you into a new view and to actually get their phone number. And I know all of you are going to call Barbara when we’re done with this, you have to actually click again. So, it’s a very immersive experience where the website and the phone number are, two to three levels down. So, from Google’s walled garden view of the world or they might put it “best experience of the consumer,” they’re creating this experience where you can explore visually and then dig in.

Carrie: Not only are they showing the earring picture related to the query, but when you click through, that earring picture is the “cover photo.” So that’s the picture you see at the top of the business list.

Mike: Correct.

Carrie: But that’s not really, like if you were to look at our business listing for a branded search, that’s not really her cover photo. So, they’re swapping those out based on the query.

Mike: This is her cover photo right next to it, right? So, they’re reinforcing the query, they’re giving the user a sense. Now this is exciting in some ways because I think consumers are very responsive to visual search. But as my wife said, “How can you pick a business based on one image?” They might have 50 kinds of pendants or 50 kinds of engagement rings. And why should Google pick which one to show, right? So, there’s a downside to it as well to some extent in the sense of it…

Carrie: I would argue though that the previous experience was it didn’t show a pendant or an earring at all. So, why should they pick? Absolutely. But at least they’re showing something related to the query versus nothing related to the query, right?

Mike: Right. Correct. Now, so we saw this last week and David and I wrote an article on it in my blog. But I would like to point out that this is not new behavior, right? A year ago, I did an analysis of these local mobile image thumbnails that we’re seeing, and at the time it became clear to me that the images that we’re seeing had no alt image tags. They had no file name relevant. That Google was embedding images based on the context of the page and the context of the image. And so, this was a year ago, it was October 3rd I wrote this article about local mobile image thumbnails, it’s all about context.

But here, too, you can see. So, this is Scanlan jewelry. You notice this as a search on engagement rings. If you have images on your page that suffice for both topics, then Google will then alternate those between them. Now in this case… Let’s see here. Oops. So, when they don’t have an image though, like…so earrings, they’re going to show potentially site links or perhaps images. Sometimes they’ll show images in the site links but we’re starting to see sometimes we’ll see images in the site links as well.

Carrie: I saw one on a Barbara result on one of these queries you were showing, for sure. That one right there.

Mike: So, jewelry repair, they’re actually showing a picture of her doing a jewelry repair and they’re cropping out from a much bigger picture. So, this ability to parse both for these thumbnails as well as this new search. And, I think it was this last August, I noticed this happening in the pack as well, right? So, again, this is jewelry. They’re showing a pendant. Here’s earrings. They’re showing earrings and here is engagement rings, and they’re showing engagement rings.

So, their understanding of imagery with no real direct clue, like no alt tags and no file names, but just on page contact, and, as we’ve learned from Andy, the image itself, right? So, there’s a tool that Andy pointed out to us and it’s at where they describe their image API and they give you the ability to drop an image in here and they show you what they understand about that image. So, this is an image from a local restaurant, The Spot To Be, that I work with on a pro bono basis. And I uploaded a lunch image for her. And they note the objects, but they also label them. So, this is clearly a dish with food, some cuisine, ingredient, brunch, lunch, a meal. And they show a likelihood of its relationship.

Carrie: What I find very, very interesting about this, Mike, is they know that this dish is brunch, lunch, breakfast. Dinner does not show up there. Not only that, they know that it’s like ham or chicken salad, right? Like if you go further down, not only did they know which meal it is, they know which type of food it is on the plate. So, it’s not just food, it’s ham salad.

Mike: That’s right. And then they relate it back to other web entities and the likelihood of it relating back to another web entity. They identify properties and what ways you might crop this differently depending on the context.

Carrie: Well, and I think those crop hints also… I mean, you’re going to show a plumbing photo later, but I think that also shows like what I called it a fraggle within an image, using Cindy’s term, what little pieces of the photo they use for their context. So not only are they looking at the photo as a whole, they’re cropping out and pulling out small pieces of the photo to get a better idea of what’s in the photo. Like that just blows my mind.

Mike: Exactly. And I took the same image and I added through Photoshop just a footer with some information. And, clearly, they were able to interpret that 100% the text on there and understand it. So, one of the things that I noticed is sometimes they identified melon as squash, as an object. And so, I specifically told them it was melon. I don’t know if they’ll be smart enough to figure that out. But I should have added cottage cheese and pineapple and grapes as well, but same crop hints on this one. So, this is an image…

Carrie: Oh, sorry. I was going to say, do you have any idea how long it takes them to like look at an image and parse that? Like if you were to change the image on your website, would it be like an instant thing? Would they figure it out right away or do you have to wait until the page gets crawled again?

Mike: Yes, it has to wait till the page gets crawled again and we don’t know how… So, we know that there’s a crawling hierarchy. The more important pages gets crawled more frequently. So, you have to get the page crawled and the page has to be rendered. Once the page is rendered, then it probably has to be passed to this API. We don’t know the priority they put on image analysis. It’s likely…like they want to deliver quick, faster results so like they’ll render HTML before they render JavaScript, as an example. They can do HTML pretty much the same day. JavaScript might take a week. I would guess that imagery is probably similarly delayed because it isn’t mission-critical. It’s just nice to know. So that would be my guess, but that’s all that is is a guess.

But notice on faces, they look at not just faces but emotions and, their degree of confidence that these are unlikely emotions. Am I likely to be sorrowful in this or angry? But I’m not likely to be joyous either. So, go figure. They did identify this is a man who’s wearing clothing. Good for them. But 99% confident that I have facial hair? And then I still have a head of hair and then I have a forehead. Those are all good things. But those are probably not good images to use in your GMB. They may convert well in terms of human interaction but in terms of them understanding what it’s doing, I don’t think it’s great.

So, I took an image from a lawyer that I work with from them sitting at a desk to see what they came up with about this lawyer. And, again, because there’s faces in it, they look for emotion. But then they analyze the objects in it. So, in here we have men and women, generic objects. Labels. Now these are interesting. They are getting close. It’s a law office, collaboration, conversation, management, employment, meeting, business person. So, the descriptions are generic.

So, Carrie suggested that we upload an image of a courtroom and see what we got. And, clearly, Google understood that it was, in fact, a courtroom. So, I think this speaks to your point about showing not just images of people but images of people doing something, right? Now here we have a very happy plumber right there. We have a 94% likelihood that this plumber is happy. He’s happy because he’s a model being paid to do stock photography. And the stock photography, just as a note, shows up all over the web in a gazillion places. So, if I were you, I would not use this stock because that would be… But we do know he’s happy.

Anyways, so the objects here that we’ve identified. But more interesting to me, again, are the labels, right? This one spot-on plumber is doing a job. He’s plumbing. Eighty-two percent, likelihood that he’s a plumber. So, I think your point about picking images of people doing the work that they’re doing is really the way to look at this for Google to understand what’s going on.

Carrie: I think this is a really valuable tool for, choosing what photos to put on a page. So, we were talking about the table with the lawyer sitting around it. Google did not pick up that there was anything to do with legal, law, lawyer, anything at that table. But they did pick it up with the courtroom photos. So, it might be hard for a lawyer to get courtroom photos, right? I mean, they’d have to stage it or whatever. But they could, and this was to your point, Mike, put a text on that page that said, employment, law, mediation, meeting or something like that. And then Google could parse the text over that photo and understand it. Maybe that would help the context like you did in the food photos. So, I think if you can’t get pictures of you doing your job or doing what you do, then the next best bet is to make sure that you have something close and you label it correctly. But I think you said 10% or less can be text?

Mike: Well, the rule in AdWords is less than 10% of the surface area can be used for this. And I think, I don’t know this for a fact, but I think they apply a similar rule in GMB. So, but you don’t need a lot of texture, right? This is just, a footer and perfectly descriptive and enables Google to better understand it. Now this image, right, I went out and I looked where it was located. And this goes…and so it found it on, a number of pages, but…plumbing by Jake. Now, it’s a good example of why Jake should go get himself a good phone and take a picture of himself fixing this sink so that it looks real, it looks authentic, and that it isn’t obvious to Google that this image is in a million other places. I mean, it’s at Yelp. It’s at Pinterest. It’s at all these other plumbers, most of which are, you know…

Carrie: I think that Google probably gives you…I mean, I don’t want to say a penalty because they don’t know it’s correct. But you might…

Carrie: I think, this is Google giving you the opportunity to see, “Hey, we know this photo is used on 400 other websites.” So, I mean, they probably are…don’t necessarily know it’s stock, but maybe they don’t trust what they see on your website as much because it’s not just on your website. I mean, you could draw that inference, I think. I mean…

Mike: Well, I don’t know that Google has, but I would suggest that image…stack imagery is a characteristic currently of spam websites. This, for example, and plumbing in Johannesburg, these are likely spammy sites. And Atlanta plumber repair, these are likely spammy sites. If Google had half a mind to, they actually could use spam imagery as a way of figuring this out. But I doubt they’ve gone this far, right? I mean, they have not demonstrated a lot of commitment to get rid of the noise in the various data sets, some of which are much more…impose a bigger burden on consumers, fake reviews, fake websites. They haven’t spent a lot of time fixing it so I doubt that they’re doing that around images. But clearly, to your point, they have the capability, which is interesting. So, I think people understand now that Google is using images, understand the images, using images to drive search behavior, using images to match user queries. And now I think the question becomes, what do we suggest people do in terms of images?

Carrie: Right, the tactics.

Mike: The tactics. And I would start with a website. Clearly, we saw in the case of Scanlan, that the homepage, which obviously I haven’t looked at it, but let’s go take a look, clearly has multiple images on it that Google is capable of understanding. Let’s go take a look at it and see what they’ve got.

Carrie: I noticed in a few 3-packs for your queries that it just had the building picture, so I’m wondering if in some cases they get it, in some cases they don’t have a picture that matches. It’d be interesting.

Mike: Yes. That is likely the case. It looks like they’ve got a bunch of different…interestingly, they have them categorized on the front page by, by necklaces and such. So, I think…and I’m not saying that this is a particular good front page. I don’t think it is. But I do think that you want to think about putting a couple of images on your front page that relate to your primary search queries in the body copy as a place to start this enterprise so that you can maximize the images that show up in the mobile thumbnails and might show up in the… Let’s go in here and see. I know that this image is actually very relevant image, right? It’s an image of jewelry repair in context. one of the things I noticed in my experiment is that galleries and header images typically do not show up as in the thumbnail areas. So, I think if you’re just using those, those aren’t enough for Google.

Carrie: So, you were saying that you think that the text around the image on the page needs to be very relevant to what the image is for Google to help Google pull that into that result, for sure.

Mike: When I did that first test a year ago, that’s what I saw. But that’s before I understood how sophisticated their AI is. So, I think it helps, but I don’t think it’s what they’re doing. Originally, when I did that test a year ago and I looked at all those mobile, the thumbnails, I thought that’s how they were determined. It was by context. But I think now, knowing what I know, that they’re probably determining it through their AI engine. And so, I don’t know that you need that much context, but I think from a user experience it makes sense that if you’re specializing in engagement rings, jewelries and repairs, and those are the three things you do, you would put images there. I think it would improve the user experience. I think it would improve Google’s understanding of who you are, what you do, and would all reinforce each other.

Carrie: And you’re saying, you’re not seeing header photos being used and that would, in my opinion, reinforce that because header photos generally don’t have a lot of text around them. The text that’s overlaid on the header photo is usually branded, for the most part. Sometimes it has keyword clues in it, but not usually. So maybe Google doesn’t trust that header photo as much as the photo with the text around it like as another clue. I think it’s very possible that that’s another context clue for them or they would be pulling header photos left and right, in my opinion.

Mike: Right. So, website, it’s a critical place for photos, critical place for Google to develop their inner entity understanding of what you do so they have not just the text now but the image. I think if you like the idea that images are showing here, I mean, why wouldn’t you want images plus site links? From my point of view, that’s much more compelling presentation than not. So, I think you would want them, I mean. And I think, as we saw with the plumber, you want to be doing real photography. You don’t want to be using stock. And then that raises the question about what do you do with the GMB?

And I think it’s a similar thing that you want to be sure that you are thinking through your product categories and uploading your best representation of those product categories in a clean, easy to understand, and easy to see at this small size image, right? I think one of the issues with mobile and with this presentation is, particularly in the pack, the image is presented in a, I think it’s 125 by 125 pixels. Notice the crop on it, right? So, it’s probably center-weighted, but it’s still like very small amount of image. Now this one is a little bit bigger but… So, what you want to do I think is if you want to get really serious, is you want to drive your images through this tool and see how it gets cropped, how Google crops it at different aspect ratios to see what they pick up as critical in the image, right? I haven’t done that…

Carrie: Do you think that that’s likely what they would show in the result if they were to show that?

Mike: That’s a great question. I don’t really know. That would take some more testing to ascertain that, but it’s certainly worth looking at that. I haven’t dug that far. This is breaking news to me as much as it is to anybody else, right?

Carrie: Sure. It would be interesting to see if that crop is how they render it in the results though. That would be an interesting test.

Mike: Right. But notice how this is cropped differently than this is cropped. Same image but cropped differently. And this one is a more vertical crop. It’s like a two to one. This is a one to one crop.

Carrie: Square versus…

Mike: Square versus rectangular. So, as we know in the local and the various presentations and various…Google is going to crop the way they want, but I think, typically, you want to work towards good center-weighted images. Let’s just see what we’ve got here. Notice how it cropped just before back here? So, I think that was the image, wasn’t it? And notice here they crop it one way and then… they cropped it a different way here and still it’s…. Now, interesting. It went back and they’re not showing it in the pack this time in the new search by photos. So that’s interesting. If I’ve rejected it once, do they not show it again here?

Carrie: Well, they do that with the search results when you bounce back and forth sometimes, so I could imagine they would do it with images.

Mike: Another interesting thought experiment on these pages is just how much of this content and how many of these links drive back in to more Google content. So, all the sponsored links at the top, the sponsored images, the view more, the pack, the various clicks on the pack, these obviously go out to the website. But then we have search by photos going out to Google. Now it has this other image one here. We have “Discover more places,” “Related searches.” I don’t know what percentage of this page is going to be driving back into Google but a fairly high percentage and a growing percentage.

Carrie: Right. All those new results we’ve seen, the “Search by photo,” the images and the three pack, etc., that’s all going to GMB. You’re not getting directly to a website for any of that stuff.

Mike: So, it points out how important it is to have all of your images in the GMB look good and for you to look at them in these various views, particularly on mobile, not on the desktop. As a note, you can do this on Chrome. Go into Chrome, go into Developer, go into Developer Tools and then you can pick the device at the lower left and then you can pick a different phone you want to use. You can pick the pixels and then the view ratio here as well. So, it’s probably easier to do it in Chrome if you’re sitting at your desk, but you want to remember to do it because most of the searches are coming that way. And then don’t forget, simultaneously, the check-in maps. We’re seeing in many categories a lot of discovery occurring in maps these days, so just search.

Carrie: Great stuff. I think it’s exciting. It’s like another frontier that local is, has the opportunity to take advantage of. And we seem to be…I’m not seeing this in straight organic search, like, discovery without location searches as much. I think that the maps results and GMB is really driving a lot of this search result, this image-based hyperlocal search.

Mike: Right. I mean, as I’ve shown in Google Trends, the increase of new researches, which is just a small percentage of the total local searches that Google is now delivering, I believe that the potential for local search with everybody having a phone in their pocket and learning new search behaviors, I think Google recognizes this and is allocating their resources to this, that local search has the potential, between maps and browser-based local search, has the potential to outstrip general search by many orders of magnitude. And I think…

Carrie: Do you think that this will show up on desktop eventually? Do you think that’s possible?

Mike: I think it is possible. I think Google often…in fact, I saw some experiments last week where we were seeing some of these…

Carrie: Image? Yeah, I saw those, too.

Mike: Thumbnail SERPs. We saw that on a desktop a little bit. So, I think Google is doing a mobile-first roll out of these features and then the ones they really like they roll back to the desktop. But I think the desktop has been moved to a lesser development environment than mobile. So, you’re going to see this stuff first on mobile going forward.

Carrie: I think that image site link is quite compelling. I think it would be quite compelling on desktop as well. It’s a little tiny on mobile but… That one right there, yeah. But on a desktop I think that could be very interesting. And so, we talk about real estate a lot. That stretches your real estate just that little bit more. That’s that one little quarter of an inch or whatever that your competitors don’t have. And, that can make a big difference on a result that, people have to scroll and scroll and scroll along to see things. So, I think anywhere where we can take advantage of real estate, we should try for sure.

Mike: Absolutely. I would love to be a fly on Google’s Analytics’ wall and see whether this result with the larger image and the straight-up site links versus this result with no image but multiple images in the site links, which one drives better conversions.

Carrie: Gosh, I wish we could see that. It’s all just going to come in as organic, unfortunately, what we can see. But to see what they can see would be so interesting. I think.

Mike: So, summary. Images are important. They’ve always been important. They’re now more important because Google can extract meaning from them in a way that could expand your reach both on your website, in organic and in all the local search results and in the ever-increasing immersive search that Google is providing users, that rabbit hole that they want people to go down. It might as well have them go down your rabbit hole rather than somebody else’s.

Carrie: For sure. I think there’s so much opportunity here and, there’s a lot of opportunity to be “first to market” with this because I think a lot of small businesses especially are not doing this right, especially if they’re in a service business where, photos might not be that important to them. Barbara, obviously, imagery is very important to what she does. So, she’s got a lot of photos on her page and the benefit of having Mike in her ear for years.

Mike: But she taught herself photography. When we first got going, her photos were terrible and we worked with her literally a couple of years to get her photo quality up and she has become very good at it. And I think, there’s no reason that you can’t with the new equipment on the market, either Pixel 3 or Pixel 4 or an iPhone 7 and up, any of those take excellent product and people shots and do a great job with lighting. And if you’re really nuts about it, get an iPhone 11.

Carrie: I heard they’re great. I’m not ready for that upgrade yet, but I just got my 10. I think this is really interesting and I’m excited to see, more testing and more opportunity and , what happens as people start implementing these new things that we’re coming up with and what the search landscape looks like as it starts to evolve. So, thanks very much for doing this, Mike. I think it’s going to be really helpful to a lot of people who don’t quite know what to do, and we’ve given them what to do to take advantage.

Mike: Thank you. One final word is that if you’re listening, this is a podcast. Come to the website and you will find the links of the articles that we’ve referred to as well as a transcription of the podcast. You can read it at your leisure. Thank you very much for listening. We appreciate it and we’ll talk to you soon.

Carrie: All right. Bye everybody.