It's pretty rude to not maintain eye contact when speaking to someone... especially when your eyes drift to an inappropriate place. Do not ignore the lady... though after this, it'll be hard to lose focus.